What is an LLM gateway?
The single endpoint every app, agent and IDE calls — and the single place you route, meter, secure and audit model traffic.
LLM gateway · AI gateway / LLM proxy / LLM router
The term is used loosely: vendors call the same layer an AI gateway, an LLM proxy or an LLM router. This page defines it precisely, separates it from the things it is confused with, and is explicit about when you should not deploy one.
What is an LLM gateway?
An LLM gateway is a service that sits between your applications and model providers. Applications call one endpoint; the gateway forwards each request to the model you configured, translating protocols, holding provider keys, and recording usage. It does not train on your traffic, rewrite your prompts, or replace the model.
- •It is infrastructure, not a model — the models stay with the providers you choose.
- •One endpoint serves chat apps, coding agents, batch jobs, realtime voice and MCP tool calls.
- •The useful dividing line: the gateway owns the request path, the application owns the prompt.
What does an LLM gateway actually do?
Four jobs: route each request to the right model, normalize protocols so any client can talk to any provider, centralize keys and access control, and record every call for cost and audit. Caching, budgets and content checks are built on top of those four, not instead of them.
- •Routing and fallback: choose a model per agent role, retry another provider on 5xx or timeout, pool upstream keys.
- •Protocol translation: four chat-protocol families in and out, so a client keeps the SDK it already ships.
- •Governance: key groups, header-based admission control, per-tenant quotas, SSO and SCIM, request-level audit logs.
- •Observability: token and cost series per key, team and model at one-minute granularity.
Why do enterprises put a gateway in front of their models?
Because model access stops being one team's problem. Once several teams, vendors and agents call models, the questions become procurement questions: who may use which model, what it costs per department, whether data left the network, and what a provider change costs in engineering time. A gateway answers them through configuration instead of code.
- •Cost attribution: every call carries a key and a tag, so spend splits by team without a reconciliation project.
- •Data path: the gateway runs in your network; requests go only to the endpoints you configure.
- •Continuity: provider outages and model changes are handled at the routing layer, not in every application.
- •Procurement: self-hosted with BYOK keeps keys and traffic under your own provider contracts.
When do you actually need an LLM gateway?
When any two of these are true: more than one model provider, more than one consuming team, a compliance or residency requirement, or spend large enough that nobody can explain the invoice. Below that line, calling provider APIs directly is simpler and cheaper — a gateway is one more service to run.
- •Two or more providers, for capability, cost or failover.
- •Per-team or per-customer cost reporting is a recurring request.
- •Data cannot leave a specific network, region or tenancy.
- •Several agents and IDEs already share one or two API keys.
When is a gateway the wrong choice?
When you have one team, one provider and one application. A gateway adds a hop, a deployment and an on-call surface. If monthly model spend is predictable and your compliance posture is simple, call the provider directly and revisit when the second provider, the second team or the first audit request arrives.
- •A prototype or a single internal tool on one provider does not need one.
- •Prompt management and model evaluation are different products — a gateway does not replace them.
- •No gateway removes the need to read your provider contracts.
How is a gateway different from a relay service or from calling provider SDKs?
A relay forwards traffic to a provider for you, often holding your key and adding a fee; a gateway you run yourself keeps keys and traffic in your network and adds routing, metering and policy. Calling provider SDKs directly skips the indirection but leaves retry, protocol and metering logic inside every application.
- •A relay can be fine for a prototype; the trade-off is that prompts and keys leave your network.
- •SDK-only is the cheapest option until the second provider arrives — then every app needs the same glue.
- •A gateway is the only option of the three that can tell you who spent what.
| Criterion | Self-hosted gateway | Relay / aggregator | Provider SDKs only |
|---|---|---|---|
| Where it runs | Your VPC or on-premises network | The vendor's cloud | Inside each application |
| Who holds the keys | Your secret store; gateway memory at runtime | The relay vendor | Each application |
| Model coverage | Any provider you configure | The vendor's catalog | One provider per SDK |
| Routing and fallback | Per role, per key, across providers | Limited to the vendor's catalog | Written by you |
| Cost reporting | Per key, group and model, from your own ledger | Vendor dashboard | Whatever you build |
| Audit surface | Request-level logs you own | Vendor logs | Scattered across apps |
Should the gateway be self-hosted or SaaS?
Self-host when keys, prompts or logs cannot leave your network, or when procurement will not accept another sub-processor. SaaS wins when you have no platform team and want zero operations. The question that decides it is not fashion: it is who can read your traffic.
- •Self-hosted costs you containers, upgrades and a runbook; SaaS costs a subscription and a sub-processor review.
- •Hybrid exists: a self-hosted data plane with a vendor control plane — the prompts may stay in, the metadata may not.
- •Ask for the data-flow diagram, not the marketing page.
| Criterion | Self-hosted | SaaS |
|---|---|---|
| Data path | Stays in your network | Passes through the vendor |
| Key custody | BYOK, in gateway memory only | Vendor-managed or forwarded |
| Operating cost | You run containers and upgrades | Vendor runs it |
| Time to first request | One Compose file | Sign-up |
| Compliance surface | Your DPA, your sub-processors | Vendor DPA and sub-processors |
| Failure domain | Your infrastructure | The vendor's status page |
How should you choose one?
Score the shortlist on five things: where it runs, what the bill anchors on, which protocols and SDKs it speaks natively, how finely it governs keys and tenants, and whether the vendor will still own the product next year. Then check the one thing no matrix shows: can you reproduce a month-old cost report today?
- •Deployment: self-hosted, SaaS or hybrid — and whether the control plane phones home.
- •Billing anchor: per instance, per seat, per token, or a percentage of spend.
- •Protocols: OpenAI, Anthropic, Gemini and DashScope in and out, with real streaming and tool-call parity.
- •Governance: key groups, header ACLs, budgets, SSO and SCIM, audit export.
- •Reproducibility: same inputs, same numbers, months later.