# GateLLM — Self-Hosted Enterprise LLM Gateway

> One-seventh the cost, same workload. A trusted, compliant, self-hosted LLM gateway (AI gateway) that orchestrates top-tier closed models alongside open SOTA models — planning on frontier models, execution on open SOTA.

Canonical: https://www.gatellm.io/

## Frontier Plans. Open SOTA Executes. One-Seventh the Cost.
Top-tier models handle planning; open SOTA models carry the routine 90%. Composition brings the bill to ~1/7 — the numbers are computed live in the calculator, not asserted here.
```
from openai import OpenAI

# One line change: point your SDK at your own gateway
client = OpenAI(
    base_url="https://your-gateway.gatellm.io/v1",
    api_key="your-gateway-api-key",
)

# Existing code unchanged — the gateway routes each request
# to the right model tier (planning vs execution) per your console policy
resp = client.chat.completions.create(
    model="claude-opus-5",  # or claude-sonnet-5, gpt-6-astra, deepseek-flash...
    messages=[{"role": "user", "content": "Plan the architecture"}],
)
```
Mix models: Claude Opus plans, DeepSeek / Qwen execute — ~1/7 the cost, up to 90% lower

The gateway decides which model runs: whatever the client is configured with, the landing model is the one admins pinned — ingress allowlist + "*" catch-all mapping

One protocol is enough: speak OpenAI or Anthropic, reach models behind all four chat families (OpenAI / Anthropic / Gemini / DashScope) — swapping models never touches your code

Data stays on-prem: fully self-hosted, prompts never reach us; credentials and PII detected on the request path — log / mask / block per scope

## Three things the gateway does that a single model cannot
Mix models to cut cost, enforce which one actually runs, and keep client code on a single protocol.
### Mix models instead of running the priciest one everywhere
Frontier models (Claude Opus) plan and reason; open SOTA models (DeepSeek / Qwen / GLM) handle routine execution. Under the default mix the bill is ~1/7, with ~97% capability retained on routine execution tasks — computed live in the calculator, not asserted here.
- Plan mode detected per request — no client changes
- Tiers assigned by key group or identity
- Per-mode cost attribution and budget caps
### The gateway decides which model actually runs
Whatever model a developer configures, the request is normalized on arrival. Names outside the allowlist get a 403; a "*" catch-all mapping rewrites everything else to the sanctioned model. Mappings and rules hot-reload — no per-developer rollout.
- Key-group allowlist gates model names at ingress
- Catch-all mapping pins the landing model
- Every request logs its real landing model (SW tag for script switches, MM for identity mappings)
### Write the client once, swap the backend freely
Your app speaks one protocol — OpenAI or Anthropic. The gateway translates across the four chat-protocol families (OpenAI / Anthropic / Gemini / DashScope) and normalizes streaming, tool calls and other capability differences, so adding a vendor or a model never touches business code.
- Any-to-any across the four chat-protocol families
- Anthropic-only clients (Claude Code, claude-code-action) can run other vendors' models
- 15 upstream protocol shapes beyond the four families, documented in the interop matrix

Integrated with 100+ LLM providers · 30+ SDKs & frameworks

## Verify it yourself
We publish the documents instead of asking you to take our word for it. Every claim on this page has a paper trail you can read before you talk to us.

### [Security & Compliance](https://www.gatellm.io/security.md)
Architecture, key handling, and isolation model.

### [Security Questionnaire](https://www.gatellm.io/trust.md)
CAIQ / SIG / VSA answers, filled in.

### [Sub-processor List](https://www.gatellm.io/subprocessors.md)
Every third party that can touch your data.

### [Incident Response & Support](https://www.gatellm.io/sla.md)
Response targets and support commitments.

## One Interface, Mix and Match All Models
From simple model routing to multi-model orchestration, GateLLM provides comprehensive enterprise-grade capabilities.
### Multi-Model Routing
One interface to call all models. Intelligent fallback, load balancing, automatic retry.
- 100+ LLM providers unified API
- Intelligent fallback routing
- Load balancing and rate limiting
- Upstream key pooling: one caller sticks to one key, fails over to the next
- Ollama / vLLM / llama.cpp as openai-compatible upstreams, one governance plane
### Composition
Tiered routing by plan mode, identity, or request shape: planning on top-tier models, execution on open SOTA. The gateway rewrites the model name — client code never changes.
- Plan-mode detection: plan on top-tier, execute on open SOTA — zero client change
- Identity-scoped model mapping: same model name → different upstreams per key group
- switch-route rules: redirect image-bearing or special requests to the right model
- In-agent downgrade: plan locked / non-plan auto-downgrade
### Multi-Model Code Review
Run 2–3 models in parallel on the same PR. The gateway supplies the protocol interop and one-key metering; the fan-out, merge, and confidence scoring live in the open-source orchestration action.
- Gateway protocol interop: non-Anthropic models join Anthropic-only review queues
- Single access key meters and governs the whole fan-out
- Fan-out, [Consensus N/M] scoring, and merge live in gatellm-io/gatellm-code-review (orchestration-side)
### MCP Tool Governance
Aggregate external MCP tool servers behind one /mcp endpoint: tool-level ACLs, context budgets, and call auditing. Which tools an agent may use lives in the same permission plane as which models it may call.
- Multiple MCP servers behind one /mcp endpoint
- Per-key-group tool allow/deny + label filters
- Built-in BM25 retrieval when the tool list grows
- REST bridge: wrap any REST API as an MCP tool from a simplified spec, zero code
- MCP calls / violations / sessions fully audited
### Web Search Injection
Give every model web search, uniformly: the gateway retrieves and injects results, synthesizing native search blocks per client protocol — open models get search grounding too.
- Native search blocks per protocol (Anthropic / OpenAI / Responses)
- Per-model three-way toggle, graceful degradation on failure
- Hijacked search excluded from usage billing
### Cost Control
Budget management, rate limiting, team cost attribution, mode-aware model policy. Never markup token fees, BYOK with your own keys.
- Cost attribution by project/feature
- Soft budget alerts
- BYOK: pay providers at standard rates
- Mode-aware model policy (plan/execute, enforceable)
### Governance & Integrations
From client access control to CI/CD integration—extend gateway capabilities into your dev and collaboration workflows.
- Client access control (User-Agent policy)
- GitHub Actions code review via your own gateway
- Batch config: stage → preview the full conflict set → single-transaction atomic commit
### Office / M365 Add-in
Use your own model gateway directly inside Excel, Word, PowerPoint & Outlook — self-hosted add-in, data never leaves the network, zero pivot dependency.
- Excel financial modeling / Word contract review / Outlook inbox
- Entra SSO + Purview label gating
- 33 bundled domain skills, paid add-on
### Transform Scripts
A sandboxed JavaScript engine (QuickJS) rewrites request and response JSON at the gateway. Edit in the console, applies live — no redeploy, no glue code.
- Before / after request slots + a response slot: see the client format pre-translation or the upstream format post-translation
- context.useModel() switches the whole route — base_url, protocol, and credentials together
- Built-in library: json_path / str_* / regex_* / array_* / encoding + hash / UUID + time
- Sandboxed (no network, no filesystem) with fail-open/closed modes and an in-console script tester

## Model Mix-and-Match: Quality No Single Model Can Deliver
Composition is a native gateway pattern; multi-model code review is an orchestration-side pattern the gateway enables — both built on the capability normalization layer.
### Composition: Tiered Routing Across Models
Route by plan mode, identity, or request shape: planning stays on top-tier models, execution drops to open SOTA. The gateway rewrites the model name and translates protocols — client code never changes.
- Plan-mode detection: route planning to top-tier, execution to open SOTA — zero client change
- Identity-scoped model mapping: same model name → different upstreams per key group / header
- switch-route rules: redirect image-bearing or special requests to the right model
- No glue code — protocol translation handled by the gateway
- Enforceable policy: admins lock model tiers per plan/execute mode, stopping all-top-tier usage
```
// before-slot transform script (JavaScript) — switch execution model by plan mode.
// Mounted on the entry model in the console; runs before routing.
function transform(body, context) {
    if (context.sourceProtocol !== "Anthropic") {
        return body;
    }
    const EXEC_MODEL = "qwen3.8-max"; // a model name configured in the gateway
    const PLAN = /plan mode is active/i;
    const EXIT = /exited plan mode/i;
    // take the last marker-bearing message: later marker wins
    const messages = body.messages || [];
    for (let i = messages.length - 1; i >= 0; i--) {
        const s = JSON.stringify(messages[i].content || "");
        const inPlan = PLAN.test(s);
        const exited = EXIT.test(s);
        if (inPlan || exited) {
            // not in plan mode → switch execution to the cost-efficient model
            if (exited || !inPlan) context.useModel(EXEC_MODEL);
            return body;
        }
    }
    context.useModel(EXEC_MODEL);
    return body;
}
```
### Multi-Model Code Review (orchestration-side)
The gateway enables the pattern, the orchestration layer runs it: claude-code-action speaks only the Anthropic protocol, and the gateway's protocol interop lets non-Anthropic models like qwen3.8-max and kimi-k3 join the review queue. A single gateway access key uniformly meters every review model.
- [Consensus N/M] confidence annotations, merge, and dedupe are produced by the action
- Each review model is billed separately and aggregated uniformly by the gateway
- PR line-count gate and historical-comment collapse are built into the action

## Three steps to your first token.
Upgrade from single-model to multi-model orchestration in three steps, no business code rewrite required. Any SDK (OpenAI / Anthropic / Gemini / DashScope) works.
### Deploy Gateway
One-click Docker launch, or Kubernetes Helm deployment to private environment.
### Configure Keys
BYOK mode: fill in your existing provider API keys, data stays on-prem, never persisted.
### Replace Endpoint
Point OpenAI SDK's base_url to GateLLM, call all models with unified interface.
```
# Just one line change: point base_url to GateLLM
from openai import OpenAI

client = OpenAI(
    base_url="https://your-gateway.gatellm.io/v1",
    api_key="your-gatellm-api-key"
)

# Existing code unchanged, call any model
response = client.chat.completions.create(
    model="claude-opus-5",  # or claude-sonnet-5, gpt-6-astra, deepseek-flash...
    messages=[{"role": "user", "content": "Hello"}]
)
```
## Or hand it to your agent

Install and configure the GateLLM gateway. First curl this install guide's raw Markdown, read it end to end, then follow each step in order (steps have dependencies — e.g. a key group must exist before an access key; don't skip ahead):

curl -fsSL https://docs.gatellm.io/en/quickstart/docker-single-node.md

The agent needs one thing from you: the upstream provider API key. It generates the rest, configures everything over the Console API — no browser — and is done when the first request returns a completion.

## Or run it yourself

```bash
docker run -d \
  --name gatellm \
  -p 7890:7890 \
  -e CONSOLE_PASSWORD=change-me-on-first-login \
  -e ENCRYPTION_KEY=REPLACE_WITH_YOUR_ENCRYPTION_KEY \
  -e CONSOLE_SECRET_KEY=REPLACE_WITH_YOUR_CONSOLE_SECRET_KEY \
  -e STORAGE_MODE=sqlite \
  -v gatellm-data:/var/lib/protoflux \
  ghcr.io/gatellm-io/gatellm:latest
```

Replace the three placeholder values before running — the guide explains what each one is and why leaving it as-is is unsafe.

## Bring the SDK you already have. Change one line.
Same four protocols on both sides. Convert any-to-any with no glue code, across chat, image, speech, video, realtime voice, embedding, and rerank.
### Every modality
- chat
- image
- speech
- video
- realtime voice
- embedding
- rerank

- Native ingress: OpenAI / Anthropic / Gemini / DashScope SDKs connect directly, no change beyond base_url
- Native egress: routes to any provider in its native protocol, not a unified OpenAI envelope re-translated
- Any-to-any conversion: ingress A → egress B seamlessly (e.g. Anthropic SDK call → routed to a DashScope-native model, converted to DashScope protocol)
- All modalities + embedding/rerank: chat / image / speech / video / realtime voice, plus embedding and rerank protocols unified
- Stronger than "OpenAI-compatible": synthorai and others offer three ingress protocols unified to one OpenAI surface; GateLLM offers four ingress + four egress, any-to-any
- Wider egress protocol surface: beyond the four major protocol families, 15 upstream protocol shapes in total (incl. AWS Bedrock Converse / Invoke, Realtime, passthrough) — enterprise models on Bedrock and self-hosted models all plug into one gateway
- Realtime voice sessions (/v1/realtime, openai_realtime + dashscope_realtime) metered per turn into the same statistics / billing / log pipeline as HTTP

## Works with the SDK you already have
Point any SDK or framework at the gateway with one line. OpenAI, Anthropic, Gemini and DashScope connect natively; LangChain, LlamaIndex, CrewAI, Vercel AI SDK and 30+ more through the same base_url swap.
### Point base_url at the gateway
Default https://your-gateway.gatellm.io — whether it includes /v1 depends on the SDK (each example gives the exact value).
### Swap in a gateway access key
Not an upstream sk-… key — the same access key mounts into whichever header your SDK expects.
### Use the gateway model name
Whatever you configured in the console — not the vendor's official name.
```
# base_url convention differs by SDK — match the one you're using
from openai import OpenAI
OpenAI(base_url="https://your-gateway.gatellm.io/v1")        # OpenAI SDK: /v1

import anthropic
anthropic.Anthropic(base_url="https://your-gateway.gatellm.io")  # Anthropic SDK: root (appends /v1/messages)

# Vercel AI SDK createAnthropic: baseURL ".../v1" (appends /messages) — the reverse
# LangChain ChatOpenAI: base_url="https://your-gateway.gatellm.io/v1"
```
### `Authorization: Bearer <key>`
- SDK / framework: OpenAI · DashScope · generic
- Bearer prefix case-insensitive

### `x-api-key: <key>`
- SDK / framework: Anthropic SDK
- Authorization also accepted

### `x-goog-api-key: <key>`
- SDK / framework: Gemini SDK
- Authorization also accepted

### Official vendor SDKs
OpenAI / Anthropic / Google SDKs connect natively — only base_url changes

### DashScope native
Text, image, async video, embedding & rerank through one endpoint

### Unified SDKs & gateways
Vercel AI SDK, LiteLLM, Portkey, Semantic Kernel, Spring AI

### Orchestration frameworks
LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, Haystack

### Agent SDKs
OpenAI Agents, Claude Agent, Pydantic AI, ADK, Mastra

### Realtime & audio
Realtime WebSocket voice + openai_audio ASR / TTS

### Local / self-hosted
Ollama, vLLM, llama.cpp as upstreams — keep any client SDK

### The model name must match the gateway config
Not the vendor's official name, not the upstream model ID — whatever you named it in the console. Wrong name → 404 model_not_found.

### Whether base_url includes /v1 varies by SDK
OpenAI SDK takes /v1; the Anthropic SDK takes the root (it appends /v1/messages); the Vercel AI SDK's createAnthropic takes /v1 (it appends /messages). Getting it wrong adds or drops a path segment.

### Fields get translated and constrained across protocols
A max_tokens may become max_output_tokens with a lower bound (e.g. ≥16) when the upstream is a Responses-protocol model — too small → 400.

### Thinking blocks are wrapped by the gateway
Claude / Gemini thinking content and signatures are passed through or stripped — take the type == "text" element of the content array, don't assume content[0] is text.

### OpenAI Agents SDK defaults to /v1/responses
It calls the Responses API, not Chat Completions. If the upstream is openai (Chat), switch explicitly with OpenAIChatCompletionsModel; if openai_response, a string model name works as-is.

### LlamaIndex validates the model name against an allowlist
The OpenAI class only accepts families it recognizes and rejects custom gateway names — use OpenAILike (Python) or declare the model info explicitly.

### CrewAI is driven by LiteLLM under the hood
base_url isn't a framework parameter — it's the OPENAI_API_BASE env var, and model needs the openai/ prefix.

### AutoGen requires explicit model_info
Unknown model names error on the default list — declare model_info explicitly to bypass the check.

### Google GenAI SDK points at the root
base_url goes via http_options.base_url (no /v1) and the key mounts in the x-goog-api-key header.

## Point any AI coding agent at the gateway
```
# Every agent is one base_url + one access key away
export ANTHROPIC_BASE_URL="https://your-gateway.gatellm.io"   # Anthropic agents: root (no /v1)
export ANTHROPIC_API_KEY="<gateway access key>"

export OPENAI_API_BASE="https://your-gateway.gatellm.io/v1"   # OpenAI-compatible agents: /v1
export OPENAI_API_KEY="<gateway access key>"

# Codex CLI (~/.codex/config.toml) speaks the Responses protocol
# [model_providers.gatellm]
# base_url = "https://your-gateway.gatellm.io/v1"
# wire_api = "responses"
```
### Terminal agents
Claude Code, Codex CLI, Gemini CLI, OpenClaw, opencode, Crush, Droid, Goose, Aider, Qwen Code, OpenHands, Kimi Code, Plandex, SWE-agent, Grok Build, Mistral Vibe, MiMo Code, ForgeCode, Codebuff

### IDEs & extensions
Cline, Roo Code, Kilo Code, Continue, Zed, Cursor, Windsurf, GitHub Copilot (BYOK)

### Managed platforms
Warp (BYOK keys only), Amazon Q Developer CLI, Devin — endpoint fixed or not overridable

### The harness pins its own model ID
Claude Code sends its configured default (e.g. claude-opus-5 or claude-sonnet-5) — not your gateway name. Either name the gateway model exactly what the harness requests, or set its model env var (e.g. ANTHROPIC_MODEL) to your gateway name. Mismatch → 404 model_not_found on the first message.

### Anthropic-protocol harnesses take the root, not /v1
The base URL for Anthropic-style clients goes to the gateway root (no /v1) — the client appends /v1/messages itself. Adding /v1 here double-stacks the path segment.

### Codex custom providers need wire_api="responses"
Codex CLI requires wire_api = "responses" for a custom [model_providers.*] block. The gateway implements /v1/responses natively — point base_url at it and set env_key to a gateway access key.

### LiteLLM-based harnesses want an openai/ model prefix
Aider, OpenHands and SWE-agent sit on LiteLLM: set OPENAI_API_BASE to the gateway and give the model an openai/ prefix (e.g. openai/gateway-model-name) so LiteLLM routes it as an OpenAI-compatible endpoint.

### Default is subscription OAuth, not your key
Codex, Gemini CLI and Qwen Code default to the vendor's OAuth / subscription login. To route through the gateway you must switch to API-key auth (set the key + base URL explicitly) — otherwise the agent bypasses the gateway entirely.

### IDEs infer tool/vision support from the model ID
Cline / Roo / Kilo and others decide tool-calling and vision capability from the model name. A custom gateway name may disable function calling — if tool calls fail, check the client's capability-detection rules or name the model to match a known family.

### Long inference adds :keep-alive and _gateway_warning
During long upstream inference the gateway emits SSE :keep-alive comment lines (harmless — skip lines starting with :), and on a mid-stream break it inserts a _gateway_warning field instead of erroring, so harnesses like Claude Code don't abort the whole conversation.

### Claude Code auto mode: decide who answers the safety verdict
In auto mode, Claude Code waits for a server-side verdict (a safeguards field in the request) before a potentially dangerous action. When the upstream is official Anthropic, the verdict is produced inside that same request and the gateway passes it through untouched. When the upstream is a third party that ignores the field, the gateway answers instead — and the shipped default (allow_all) releases every action, which is more permissive than leaving the call alone. Hand a route back to its upstream by writing "anthropic_server_safeguards": "upstream" in that upstream's extra_config; the console setting is admin-only.

## Capability Normalization: Making Cross-Vendor Mixing Possible
Cross-vendor model mixing rests on the capability normalization layer, which unifies seven coupling points so switching models never means rewriting glue code.
### Tool Calling Normalization
OpenAI / Anthropic / Gemini / Chinese models all have different tool calling schemas. We unify them at the foundation, write code once, call all models.
### Structured Output Compatibility
Each provider uses different protocols (strict JSON schema, json_mode, tool_use). We unify to a consistent interface.
### Prompt Caching Unification
Anthropic cache_control, OpenAI auto-caching, Gemini context—each has different mechanisms. We provide unified caching strategy.
### Reasoning Token Adaptation
o1 / Claude thinking / DeepSeek R1 output formats differ. We adapt to each provider's reasoning token protocol.
### Multimodal Format Support
vision / audio / file upload protocols differ across providers. We unify multimodal input formats.
### Billing Model Abstraction
Each provider's pricing structure differs. We abstract to a unified cost tracking interface.
### Error-Semantics Normalization
Each vendor's error codes and bodies differ. We normalize to a uniform error body, with z-request-id correlating every request end-to-end.
### Data Flow: Stays On-Prem, Never Persisted
In BYOK mode, API keys are stored only in gateway memory, never persisted or transmitted. Requests route through gateway directly to provider official APIs, data never leaves your private network.
SOC 2 controls aligned

ISO 27001 controls aligned

GDPR

## Why ~1/7 the Cost?
Take an enterprise running 1B tokens a day: same workload — compare the bill for "top-tier end to end" versus "top-tier plans, open models execute."
## Three Sources of Savings:
- BYOK no token markup—you pay providers at standard rates, GateLLM takes no middle cut (aggregators typically add platform and payment fees on top of list price);
- Composition—open SOTA models (GLM / Qwen / DeepSeek) carry the bulk of execution (classification, extraction, translation), top-tier closed models (like Claude Opus) only for planning and hard reasoning;
- Prompt caching—high-frequency calls hit cache, unit price drops 50%–90%.
Figures are illustrative models based on public pricing, actual savings depend on business scenarios and mix ratios. Pro license fee of $400/mo (2GB single instance) is included in GateLLM plan.

## Usage you can see, cost you can settle, budgets you can hold
Token Ops tracks who uses what; Token FinOps turns tokens into chargeable, capped money. One control plane makes AI spend as operable as cloud spend.
### Usage & Cost Observability
Break every call down by model, access key, and key group, with input / output / cache-hit tokens metered separately at 1-minute granularity.
- Grouped by model / key / key group
- Input, output, and cache-hit metered separately
- 24h / 7d / 30d / custom time series
### Client Admission & Audit Trail
Only clients you have approved reach the gateway — allow, throttle, or deny by User-Agent. Every call leaves an auditable record for endpoint governance and compliance evidence.
- Client access control (User-Agent policy)
- Call records with log retention (Pro and up)
- Audit / DLP integration (Enterprise)
### Management API (programmatic control)
Manage the gateway via API: create / revoke API keys, set quotas and budgets, query usage and cost, export audit logs — wire it into your internal ops and billing systems.
- Programmatic API key create / revoke
- Quotas and budgets via API
- Usage and cost query API
- Audit log export
### Your Own Rate Card & Price Snapshots
Enter the private price you negotiated by hand and it becomes authoritative — system refreshes never overwrite it. Each price carries an effective window: schedule price cuts ahead, and edits never rewrite historical cost.
- Pinned private price shadows auto, never overwritten
- Snapshots take effect by time window
- Backfill history / pre-schedule price cuts
- Token · per-call · per-second billing modes
### Tag-Based Allocation & Chargeback
Tag keys and key groups, then aggregate usage and cost by tag to land token spend on the right cost center. Excel export splits the ledger by model × key for internal showback or chargeback.
- Free-form tags on keys / key groups
- Aggregate usage and cost by tag
- Excel export with 6 sheets (incl. key × model)
- Ready for internal showback / chargeback
### Cost Quotas & Limits
Set daily and monthly spend caps per key — judged independently on UTC calendar day and month, blocked the moment either trips. Quotas inherit key → group → global, taking the widest across groups.
- Daily / monthly caps, judged independently
- Key → group → global inheritance
- 429 quota_exceeded, Retry-After on daily

## GateLLM vs LiteLLM
A source-verified comparison. GateLLM alone ships several mechanisms LiteLLM does not have; both share a settled baseline; a few things differ in kind. Decide on the top two layers, not on marketing noise.
### Only GateLLM does this
Mechanisms LiteLLM does not have — the one-line "how it works" sits under each capability.
| Capability | GateLLM | LiteLLM |
| --- | --- | --- |
| Conditional model switching | ✓ | ✗ |
| Cross-protocol model / SDK mixing | ✓ | ✗ |
| Single-transaction config commit | ✓ | ✗ |
| Cost quota hard gate | ✓ | ✗ |
| Crash-safe billing | ✓ | ✗ |
| Upstream key pooling & sticky binding | ✓ | ✗ |
| Three billing modes + multi-currency | ✓ | ✗ |
| Upstream SSO credential login | ✓ | ✗ |
| Capability metadata registry | ✓ | ✗ |
| LTS release branch | ✓ | ✗ |

- **Conditional model switching**: Declarative switch-route rules + built-in plan-mode detection + context.useModel() scripts — zero client change. LiteLLM needs custom Python callbacks.
- **Cross-protocol model / SDK mixing**: Native any-to-any entrypoints — call GPT from the Anthropic SDK, DeepSeek from the Gemini SDK. LiteLLM is OpenAI-centric; reverse paths go through shims.
- **Single-transaction config commit**: Stage → preview the full conflict set → atomic commit; the whole batch rolls back on any conflict.
- **Cost quota hard gate**: Daily / monthly cycles return 429 quota_exceeded the moment spend trips. LiteLLM sends budget alerts only.
- **Crash-safe billing**: Pre-deduct → settle → refund with an in-flight ledger — correct across process crashes.
- **Upstream key pooling & sticky binding**: Weighted FNV-1a hashing pins each caller to one key, fails over to the next untried key; weight=0 standby.
- **Three billing modes + multi-currency**: Token / per-call / per-duration, USD/CNY/EUR/JPY/GBP, custom dimensions. LiteLLM is token-based tracking only.
- **Upstream SSO credential login**: Onboards vendors without static keys (e.g. Kiro). LiteLLM takes static API keys only.
- **Capability metadata registry**: Drives auto-injection & degradation; new-model capability drift needs no code change.
- **LTS release branch**: N-1 + LTS branch (see /lifecycle). LiteLLM supports the last 4 minor lines only.

### Both do these well
A settled baseline — either gateway covers this layer. The decision lives in the layers above and below.
| Capability | GateLLM | LiteLLM |
| --- | --- | --- |
| API routing · fallback · load balancing | ✓ | ✓ |
| BYOK, no token markup | ✓ | ✓ |
| Fully self-hosted, data stays on-prem | ✓ | ✓ |
| Cross-vendor tool calling normalization | ✓ | ✓ |
| Structured output (JSON Schema) normalization | ✓ | ✓ |
| Reasoning / thinking token adaptation | ✓ | ✓ |
| Cross-model prompt caching | ✓ | ✓ |
| Web search injection | ✓ | ✓ |
| MCP gateway (tool aggregation) | ✓ | ✓ |
| Visual console for configuration | ✓ | ✓ |
| Audit logs & usage / cost metering | ✓ | ✓ |

### Different in kind
Not better or worse — different mechanisms and packaging you should weigh directly.
| Capability | GateLLM | LiteLLM |
| --- | --- | --- |
| Runtime | Single Rust binary · no GC pauses | Python process + dependency stack |
| Protocol translation surface | 15 protocol shapes; the four chat-protocol families (OpenAI / Anthropic / Gemini / DashScope) translate any-to-any, the rest per interop matrix | OpenAI-centric envelope + compatible endpoints |
| Native protocol entrypoints | /v1/messages + count_tokens, Gemini generateContent, /v1beta/cachedContents CRUD, /v1/realtime, /mcp — one gateway serves every SDK natively | OpenAI-compatible endpoint (plus compatible variants) |
| Prompt caching depth | Auto-injects cache breakpoints on OpenAI-protocol calls + cross-protocol mapping + Google cachedContent dedup | Passes through cache_control, normalizes usage |
| Log privacy defaults | Request logs off by default · stored log bodies redacted at every egress surface, raw reveal admin-only and audited · 7-day default retention | Request/response logging on; redaction via guardrails |
| Configuration mode | Fully in console — no YAML, changes apply live | config.yaml-centric |
| Enterprise SSO & SCIM packaging | OIDC + SAML 2.0 + SCIM bundled in the flat per-instance license, no per-seat fees | OSS caps SSO at 5 users; SCIM requires the usage-priced Enterprise tier |
| Licensing | Flat per instance / memory, no seats, no token markup | Free OSS; Enterprise is usage-priced |
| Support SLA | Priority support included in the flat license | Standard tier has no response-time commitment; 24/7 SLA is a paid add-on (Sev0 1h / Sev1 6h) |
| Endpoint surface | Chat / realtime / MCP core endpoints | Adds /batches, /files, /fine_tuning, /vector_stores, /videos, /containers |
| Third-party ecosystem | Built-in logs / Prometheus / OTel — standard-protocol egress, no single-vendor lock-in; secret & PII detection built into the gateway, no third-party engine required | AWS KMS / Vault / CyberArk secret managers; Presidio PII guardrails; Langfuse / Langsmith / Arize observability |

Verified against each product's official docs as of 2026-09-04 (docs.litellm.ai / docs.gatellm.io). Some LiteLLM capabilities require an Enterprise license; details can drift — check the sources before procurement.

## Secrets and PII are detected on the way in — you decide what happens next.
A deterministic rule engine inspects every model request for credential formats and PII, and records what it found without recording the secret itself. The built-in rules ship switched off and in observe mode; you arm them and choose log, mask or block per scope. No model sits in the detection path.
### Deterministic detection, no model in the path
Deterministic by construction: literal probes first, then regex rules that require context words, then checksum validators — Luhn for card numbers, mod-97 for IBAN, the national-ID checksum, PEM and OpenSSH private-key blocks, IPv4.
- Credential formats: model-provider API keys, cloud access keys, GitHub and GitLab tokens, JWTs, PEM and OpenSSH private keys, Slack tokens
- PII classes: mainland-China mobile numbers, national resident IDs, bank card numbers, IBANs, email addresses, IPv4 addresses
- Multi-turn conversations scan only what is new — a cursor hash chain keeps a long session from re-scanning text it already cleared
### Masking your client can see through; blocking that leaks nothing
Under mask, the upstream model receives numbered placeholders and your client gets the real values restored — including per-SSE-frame restoration while streaming. Under block, the refusal carries the rule category and a request id, never the matched text.
- The upstream sees placeholders; the party that owns the data sees its own values come back
- Long pastes are not exempt — oversized strings are spilled to disk and scanned back in fixed-size windows, so memory stays flat regardless of payload size
- The allowlist matches by value, never by path: approving a test key does not quietly exempt a whole field
### Log bodies: stored as received, redacted at every exit
Request-log persistence is off in the shipped image. Switch it on and bodies are stored as received, then redacted at every egress surface — the Console API redacts by default, SSE and terminal output are always redacted, and a redaction failure closes the response instead of leaking through it.
- Revealing raw content is an explicit act: an admin role plus an explicit reveal flag, audited and counted
- Redaction covers sensitive body keys, sensitive query and URL parameters, static and custom auth-scheme headers, and bearer-token values with partial reveal
- A finding keeps a masked snippet with bounded surrounding context, never the matched text; findings share the log-retention window, 7 days by default
Off by default, on purpose
The built-in rule pack ships disabled, and every built-in rule action is log. That is the safe rollout rather than a gap: see what would have happened across your real traffic, tune the allowlist against your own false positives, then arm mask or block where it matters. Policy scopes to global, per access-key group, or per access key, and rules hot-reload.
#### log
Observe. Findings land in the console with masked snippets; the request itself is untouched.
#### mask
Enforce, reversibly. The upstream sees placeholders; your client still receives the real value.
#### block
Refuse. The request is rejected with the rule category and a request id — never the matched text.
### What it is, and what it is not
#### What it is
- Runs inside your own gateway process, on the model-request path, scoped globally / per access-key group / per access key
- A dedicated console page: a findings view for admin and monitor roles, a rules view for admin, an on/off switch with a memory-budget pre-check, hot reload, and audit entries
- A dry-run preview endpoint, so a rule can be tested against a sample before it is armed
- Every finding traces to a rule name, category, severity, action taken, match count, model and access-key name

#### What it is not
- No response-content detection — the response leg only restores placeholders
- No ML, no jailbreak or prompt-injection detection and no moderation API: the rules are deterministic
- No offline re-audit — content that was not scanned at request time is not scanned later
- Not a DLP suite: no endpoint, email or other egress-channel coverage
- No passport, SSN, name or address detectors, and known residual gaps are documented — base64-encoded media leaves are skipped, and a placeholder split across two stream frames passes through unaltered

## What you actually get.
The three things procurement cares about, settled before the price: a DPA legal can sign, a bill that can't run away, and prompts that never reach us.
### A DPA your legal team can sign
GDPR and CCPA ready, with a DPA your legal team can actually sign. Self-hosted — data never leaves your perimeter — with Enterprise SSO (OIDC / SAML 2.0) and SCIM 2.0 provisioning included.
- GDPR
- CCPA
- DPA
- SSO
- SCIM
- Audit trail
### A bill that can't run away
BYOK: pay providers at standard rates — no token markup, no per-seat fees. Hard quota caps per key and project keep a runaway script from outrunning your budget.
- No token markup
- 0 per-seat fees
- Prompt cache
- Hard quota caps
- 30-day refund
### Your prompts never reach us
Fully self-hosted — your prompts and completions never enter our systems at all. BYOK keys live only in gateway memory; requests go straight to provider APIs, never leaving your network.
- Data stays on-prem
- Keys never persisted
- TLS 1.3
- Client admission
- Self-selected models

## Fully Self-Hosted · Single Instance 2GB · Never Markup Token Fees
BYOK (bring your own keys) — you pay providers at standard rates. We license a single 2GB instance, never by seats or token markup. Larger memory or multi-instance goes through Enterprise.
The core is built on Rust. At the gateway-forwarding level, each 1GB of memory sustains roughly 100M tokens per minute (TPM) and ~500 requests per minute (RPM), comfortably serving ~100 typical users. These are reference figures: actual throughput is bounded by your own providers’ rate limits and request complexity, and is not an SLA commitment.
Included on every plan: BYOK with zero token markup · 100+ provider routing · client access control · prompt caching · cost quotas
### Free
$0
Fully self-hosted · Single license
512MB single instance, no log retention — right for evaluation and POCs, not production traffic.
- 100+ LLM routing · Fallback · Load balancing
- Cross-vendor model mixing · in-agent downgrade (plan locked / non-plan auto-downgrade)
- Client access control (User-Agent policy)
- Prompt caching
- BYOK (bring your own keys, no markup)
- 512MB memory cap · SQLite · single instance
- GitHub Actions code review via your own gateway
- No log retention
### Pro
$400
Fixed 2GB cap, no add-on. For multi-node, compliance (Audit/DLP), or the Management API, go Enterprise.
- All Free features
- 2GB memory · single instance (fixed cap, no add-on)
- Log retention
- Enterprise SSO (OIDC: Azure AD / Okta / generic, plus SAML 2.0 + SCIM)
- PostgreSQL · 2GB memory
- Email / ticket support
100% refund within 30 days if unsatisfied
### Enterprise
from $400/GB
By memory capacity · Billed annually · 30-day refund
- All Pro features
- Capacity-based billing (nodes × memory, flat $400/GB/mo)
- Multi-node / cluster deployment
- GitHub Actions code review via your own gateway
- Microsoft 365 add-in (Excel/Word/PPT/Outlook, paid add-on)
- Extended log retention
- Priority support response
- Compliance (Audit / DLP)
- Management API (programmatic API key · quota · usage query · audit export)
Need more capacity or a custom setup?
## Feature Comparison

| Capability | Free | Pro | Enterprise |
| --- | --- | --- | --- |
| Multi-Model Routing · Fallback · Load Balancing | ✓ | ✓ | ✓ |
| Cross-Vendor Model Mixing (Planning/Execution by Vendor) | ✓ | ✓ | ✓ |
| In-Agent Model Downgrade (plan locked / non-plan auto-downgrade) | ✓ | ✓ | ✓ |
| Prompt Caching | ✓ | ✓ | ✓ |
| BYOK (Bring Your Own Keys, No Markup) | ✓ | ✓ | ✓ |
| Memory Tier | 512MB | 2GB (fixed) | Custom |
| Storage / Database | SQLite (single instance) | PostgreSQL · 2GB | PostgreSQL + Redis Cluster |
| Deployment Scale | 1 instance | 1 instance · 2GB | Multi-Node / Cluster |
| Throughput per GB (reference) | — | 2GB ≈ 200M TPM / 1,000 RPM | ≈ 100M TPM / 500 RPM per GB |
| Log Retention | None | Retained | Extended Retention |
| SSO (OIDC / SAML 2.0) | — | ✓ | ✓ |
| SCIM 2.0 auto-provisioning | — | ✓ | ✓ |
| Compliance (Audit / DLP) | — | — | ✓ |
| Management API (programmatic key · quota · usage · audit) | — | — | ✓ |
| Self-owned rate card / price snapshots (incl. private pricing) | ✓ | ✓ | ✓ |
| Cost quotas (daily / monthly, three-level inheritance) | ✓ | ✓ | ✓ |
| Support Response | Community | Email / Ticket | Priority Response |
| Mode-Aware Model Policy (plan/execute, enforceable) | ✓ | ✓ | ✓ |
| Client Access Control (User-Agent policy) | ✓ | ✓ | ✓ |
| GitHub Actions Code Review (via your gateway) | ✓ | ✓ | ✓ |
| Microsoft 365 Add-in (Excel/Word/PPT/Outlook, self-hosted) | — | Add-on | Add-on |
| Purview sensitivity-label gating (addin.access / file.upload) | — | — | ✓ |

## FAQ
### How much latency does an AI gateway add?

A SaaS gateway adds an extra hop out of your network. GateLLM is self-hosted inside your VPC; the extra hop to the vendor API is local inter-process communication — sub-millisecond overhead, with negligible impact on streaming time-to-first-token.
### What happens if a provider fails or hits a rate limit?

Direct-to-one-provider is a single point of failure. GateLLM supports cross-provider fallback — when the primary returns 5xx or times out, traffic moves to a backup provider with no code changes. It also pools multiple keys per upstream: a deterministic hash stickily binds each caller to one key, and when a key fails the gateway retries with the next untried key.
### Do I actually need a gateway?

Honest bar: single provider, single team, no compliance constraint, manageable monthly spend — call APIs directly; you do not need a gateway. A gateway starts paying for itself when you need multi-provider failover, per-team cost attribution, compliance / data residency, or real-time visibility into runaway spend.
### How does GateLLM make money in BYOK mode?

We license a single instance with 2GB memory (Pro $400/mo, annual 20% off at $3,840/yr), no token markup, no per-seat fees. You pay providers at standard rates, GateLLM takes no middle cut.
### Will my API keys and data leave my network?

No. In BYOK mode, API keys are stored only in gateway memory, never persisted or transmitted. Requests route through the gateway directly to provider official APIs — data never leaves your private network.
### How do I know the gateway serves the real model, not a cheaper substitute?

With a SaaS aggregator you can only trust its word. GateLLM is BYOK and self-hosted: you use your own keys and requests go from your gateway straight to the vendor API endpoint — model identity is guaranteed by the vendor API itself, with no replaceable hop in between.

## Start Building Your Multi-Model AI Apps
Fully self-hosted · Single instance 2GB · Never markup token fees.

## Related
- [Pricing](https://www.gatellm.io/pricing.md)
- [Cost Calculator](https://www.gatellm.io/calculator.md)
- [About](https://www.gatellm.io/about.md)
- [Book a Demo](https://www.gatellm.io/demo.md)
- [FAQ](https://www.gatellm.io/faq.md)
- [Compare AI Gateways](https://www.gatellm.io/compare.md)
- [Compare LiteLLM](https://www.gatellm.io/compare/litellm.md)
- [Compare OpenRouter](https://www.gatellm.io/compare/openrouter.md)
- [Compare Alibaba Cloud AI Gateway](https://www.gatellm.io/compare/alibaba-cloud.md)
- [Compare Azure API Management](https://www.gatellm.io/compare/azure-apim.md)
- [Compare Apigee](https://www.gatellm.io/compare/apigee.md)
- [Compare Cloudflare AI Gateway](https://www.gatellm.io/compare/cloudflare.md)
- [Compare Kong AI Gateway](https://www.gatellm.io/compare/kong.md)
- [Compare Portkey](https://www.gatellm.io/compare/portkey.md)
- [LiteLLM vs Kong AI Gateway](https://www.gatellm.io/compare/litellm-vs-kong.md)
- [LiteLLM vs OpenRouter](https://www.gatellm.io/compare/litellm-vs-openrouter.md)
- [LiteLLM vs Portkey](https://www.gatellm.io/compare/litellm-vs-portkey.md)
- [Use Cases](https://www.gatellm.io/use-cases.md)
- [Reduce LLM Cost](https://www.gatellm.io/use-cases/reduce-cost.md)
- [High Availability](https://www.gatellm.io/use-cases/high-availability.md)
- [Multi-Tenant Isolation](https://www.gatellm.io/use-cases/multi-tenant.md)
- [Compliance & Audit](https://www.gatellm.io/use-cases/compliance-audit.md)
- [Troubleshooting by Symptom](https://www.gatellm.io/use-cases/troubleshooting.md)
- [Mixed Multi-Vendor Model Stack](https://www.gatellm.io/use-cases/mixed-model-stack.md)
- [Self-Host AI Inside Office](https://www.gatellm.io/use-cases/self-hosted-office-ai.md)
- [Token Usage Visibility & Attribution](https://www.gatellm.io/use-cases/token-ops.md)
- [Token FinOps: AI Cost Allocation & Control](https://www.gatellm.io/use-cases/token-finops.md)
- [MCP Tool Governance](https://www.gatellm.io/use-cases/mcp-tool-governance.md)
- [Web Search for Every Model](https://www.gatellm.io/use-cases/web-search-grounding.md)
- [Multi-Key Rotation & Standby](https://www.gatellm.io/use-cases/multi-key-pooling.md)
- [Admission Control for Connected Clients](https://www.gatellm.io/use-cases/client-admission-control.md)
- [Enforce Which Model Agents Actually Run](https://www.gatellm.io/use-cases/enforce-model-policy.md)
- [No Model Choice on the Client](https://www.gatellm.io/use-cases/automatic-model-tiering.md)
- [One Protocol, Any Model](https://www.gatellm.io/use-cases/one-protocol-any-model.md)
- [Office / Microsoft 365 Add-in](https://www.gatellm.io/office-addin.md)
- [Glossary](https://www.gatellm.io/glossary.md)
- [Model Catalog](https://www.gatellm.io/models.md)
- [SDK & Framework Integrations](https://www.gatellm.io/integrations.md)
- [AI Coding Agents & Harnesses](https://www.gatellm.io/coding-agents.md)
- [Security & Compliance](https://www.gatellm.io/security.md)
- [Privacy Policy](https://www.gatellm.io/privacy-policy.md)
- [Terms of Service](https://www.gatellm.io/terms.md)
- [Cookie Policy](https://www.gatellm.io/cookies.md)
- [Data Processing Agreement (DPA)](https://www.gatellm.io/dpa.md)
- [Sub-processor List](https://www.gatellm.io/subprocessors.md)
- [Security Questionnaire (CAIQ/SIG/VSA)](https://www.gatellm.io/trust.md)
- [Incident Response & Support](https://www.gatellm.io/sla.md)
- [Vulnerability Disclosure Policy](https://www.gatellm.io/vulnerability-disclosure.md)
- [Open Source License Notice](https://www.gatellm.io/open-source-licenses.md)
- [Version Support & Lifecycle (EOL)](https://www.gatellm.io/lifecycle.md)
- [Engineering Blog](https://www.gatellm.io/blog.md)
- [Your Agents Have More Permissions Than Your Employees: Governing MCP Tools at the Gateway](https://www.gatellm.io/blog/mcp-tool-governance.md)
- [The $10k Invoice Problem: Five Ways to Cut LLM Spend Without Dumbing Down Your App](https://www.gatellm.io/blog/reduce-cost.md)
- [Your Vendor's List Price Is Not Your Cost: Applying Real FinOps to Token Spend](https://www.gatellm.io/blog/token-finops.md)
- [llms.txt](https://www.gatellm.io/llms.txt)