# What Is an LLM Gateway? Definition, Uses & How to Choose - GateLLM

> An LLM gateway routes every application, agent and IDE through one endpoint: protocol translation, key custody, metering and audit. What it does, when you need one, when you do not, and how to choose.

Canonical: https://www.gatellm.io/llm-gateway/

## What is an LLM gateway?

The single endpoint every app, agent and IDE calls — and the single place you route, meter, secure and audit model traffic.

The term is used loosely: vendors call the same layer an AI gateway, an LLM proxy or an LLM router. This page defines it precisely, separates it from the things it is confused with, and is explicit about when you should not deploy one.

LLM gateway — AI gateway / LLM proxy / LLM router

## What is an LLM gateway?

An LLM gateway is a service that sits between your applications and model providers. Applications call one endpoint; the gateway forwards each request to the model you configured, translating protocols, holding provider keys, and recording usage. It does not train on your traffic, rewrite your prompts, or replace the model.

- It is infrastructure, not a model — the models stay with the providers you choose.
- One endpoint serves chat apps, coding agents, batch jobs, realtime voice and MCP tool calls.
- The useful dividing line: the gateway owns the request path, the application owns the prompt.

## What does an LLM gateway actually do?

Four jobs: route each request to the right model, normalize protocols so any client can talk to any provider, centralize keys and access control, and record every call for cost and audit. Caching, budgets and content checks are built on top of those four, not instead of them.

- Routing and fallback: choose a model per agent role, retry another provider on 5xx or timeout, pool upstream keys.
- Protocol translation: four chat-protocol families in and out, so a client keeps the SDK it already ships.
- Governance: key groups, header-based admission control, per-tenant quotas, SSO and SCIM, request-level audit logs.
- Observability: token and cost series per key, team and model at one-minute granularity.

## Why do enterprises put a gateway in front of their models?

Because model access stops being one team's problem. Once several teams, vendors and agents call models, the questions become procurement questions: who may use which model, what it costs per department, whether data left the network, and what a provider change costs in engineering time. A gateway answers them through configuration instead of code.

- Cost attribution: every call carries a key and a tag, so spend splits by team without a reconciliation project.
- Data path: the gateway runs in your network; requests go only to the endpoints you configure.
- Continuity: provider outages and model changes are handled at the routing layer, not in every application.
- Procurement: self-hosted with BYOK keeps keys and traffic under your own provider contracts.

## When do you actually need an LLM gateway?

When any two of these are true: more than one model provider, more than one consuming team, a compliance or residency requirement, or spend large enough that nobody can explain the invoice. Below that line, calling provider APIs directly is simpler and cheaper — a gateway is one more service to run.

- Two or more providers, for capability, cost or failover.
- Per-team or per-customer cost reporting is a recurring request.
- Data cannot leave a specific network, region or tenancy.
- Several agents and IDEs already share one or two API keys.

## When is a gateway the wrong choice?

When you have one team, one provider and one application. A gateway adds a hop, a deployment and an on-call surface. If monthly model spend is predictable and your compliance posture is simple, call the provider directly and revisit when the second provider, the second team or the first audit request arrives.

- A prototype or a single internal tool on one provider does not need one.
- Prompt management and model evaluation are different products — a gateway does not replace them.
- No gateway removes the need to read your provider contracts.

## How is a gateway different from a relay service or from calling provider SDKs?

A relay forwards traffic to a provider for you, often holding your key and adding a fee; a gateway you run yourself keeps keys and traffic in your network and adds routing, metering and policy. Calling provider SDKs directly skips the indirection but leaves retry, protocol and metering logic inside every application.

- A relay can be fine for a prototype; the trade-off is that prompts and keys leave your network.
- SDK-only is the cheapest option until the second provider arrives — then every app needs the same glue.
- A gateway is the only option of the three that can tell you who spent what.

| Criterion | Self-hosted gateway | Relay / aggregator | Provider SDKs only |
| --- | --- | --- | --- |
| Where it runs | Your VPC or on-premises network | The vendor's cloud | Inside each application |
| Who holds the keys | Your secret store; gateway memory at runtime | The relay vendor | Each application |
| Model coverage | Any provider you configure | The vendor's catalog | One provider per SDK |
| Routing and fallback | Per role, per key, across providers | Limited to the vendor's catalog | Written by you |
| Cost reporting | Per key, group and model, from your own ledger | Vendor dashboard | Whatever you build |
| Audit surface | Request-level logs you own | Vendor logs | Scattered across apps |

## Should the gateway be self-hosted or SaaS?

Self-host when keys, prompts or logs cannot leave your network, or when procurement will not accept another sub-processor. SaaS wins when you have no platform team and want zero operations. The question that decides it is not fashion: it is who can read your traffic.

- Self-hosted costs you containers, upgrades and a runbook; SaaS costs a subscription and a sub-processor review.
- Hybrid exists: a self-hosted data plane with a vendor control plane — the prompts may stay in, the metadata may not.
- Ask for the data-flow diagram, not the marketing page.

| Criterion | Self-hosted | SaaS |
| --- | --- | --- |
| Data path | Stays in your network | Passes through the vendor |
| Key custody | BYOK, in gateway memory only | Vendor-managed or forwarded |
| Operating cost | You run containers and upgrades | Vendor runs it |
| Time to first request | One Compose file | Sign-up |
| Compliance surface | Your DPA, your sub-processors | Vendor DPA and sub-processors |
| Failure domain | Your infrastructure | The vendor's status page |

## How should you choose one?

Score the shortlist on five things: where it runs, what the bill anchors on, which protocols and SDKs it speaks natively, how finely it governs keys and tenants, and whether the vendor will still own the product next year. Then check the one thing no matrix shows: can you reproduce a month-old cost report today?

- Deployment: self-hosted, SaaS or hybrid — and whether the control plane phones home.
- Billing anchor: per instance, per seat, per token, or a percentage of spend.
- Protocols: OpenAI, Anthropic, Gemini and DashScope in and out, with real streaming and tool-call parity.
- Governance: key groups, header ACLs, budgets, SSO and SCIM, audit export.
- Reproducibility: same inputs, same numbers, months later.

## LLM gateway FAQ

### Is an LLM gateway the same as an AI gateway?

In practice, yes. Both names describe the layer between applications and models. Vendors differ in surface area: some also proxy images, audio or embeddings. GateLLM covers chat, realtime voice, web search and MCP tool calls through the same endpoint.

### Does a gateway add latency?

One network hop inside your own network. GateLLM adds no inference step of its own; the measured overhead is small next to the upstream provider response, and streaming tokens are passed through as they arrive.

### Does the gateway see my prompts?

It processes them in memory to route them and, if you enable it, to redact credentials and PII on the request path. Request logs are off by default; when enabled, stored bodies are redacted at every egress point and revealing raw content is admin-only and audited.

### Can one gateway serve several teams?

Yes. Key groups, per-group ACLs and per-tenant quotas are the mechanism: each team gets its own keys and budget while the model catalog and routing rules stay shared.

## Keep reading

- [Compare AI Gateways](https://www.gatellm.io/compare.md)
- [Use Cases](https://www.gatellm.io/use-cases.md)
- [Glossary](https://www.gatellm.io/glossary.md)
- [Cost Calculator](https://www.gatellm.io/calculator.md)
- [Cost Comparison Methodology](https://www.gatellm.io/methodology.md)
- [Enterprise AI Governance](https://www.gatellm.io/enterprise-ai-governance.md)

## Related
- [Home](https://www.gatellm.io/index.md)
- [Pricing](https://www.gatellm.io/pricing.md)
- [Cost Calculator](https://www.gatellm.io/calculator.md)
- [About](https://www.gatellm.io/about.md)
- [Book a Demo](https://www.gatellm.io/demo.md)
- [FAQ](https://www.gatellm.io/faq.md)
- [Compare AI Gateways](https://www.gatellm.io/compare.md)
- [Compare LiteLLM](https://www.gatellm.io/compare/litellm.md)
- [Compare OpenRouter](https://www.gatellm.io/compare/openrouter.md)
- [Compare Alibaba Cloud AI Gateway](https://www.gatellm.io/compare/alibaba-cloud.md)
- [Compare Azure API Management](https://www.gatellm.io/compare/azure-apim.md)
- [Compare Apigee](https://www.gatellm.io/compare/apigee.md)
- [Compare Cloudflare AI Gateway](https://www.gatellm.io/compare/cloudflare.md)
- [Compare Kong AI Gateway](https://www.gatellm.io/compare/kong.md)
- [Compare Portkey](https://www.gatellm.io/compare/portkey.md)
- [Microsoft 365 Copilot Alternative](https://www.gatellm.io/compare/copilot.md)
- [LiteLLM vs Kong AI Gateway](https://www.gatellm.io/compare/litellm-vs-kong.md)
- [LiteLLM vs OpenRouter](https://www.gatellm.io/compare/litellm-vs-openrouter.md)
- [LiteLLM vs Portkey](https://www.gatellm.io/compare/litellm-vs-portkey.md)
- [Use Cases](https://www.gatellm.io/use-cases.md)
- [Reduce LLM Cost](https://www.gatellm.io/use-cases/reduce-cost.md)
- [High Availability](https://www.gatellm.io/use-cases/high-availability.md)
- [Multi-Tenant Isolation](https://www.gatellm.io/use-cases/multi-tenant.md)
- [Compliance & Audit](https://www.gatellm.io/use-cases/compliance-audit.md)
- [Troubleshooting by Symptom](https://www.gatellm.io/use-cases/troubleshooting.md)
- [Mixed Multi-Vendor Model Stack](https://www.gatellm.io/use-cases/mixed-model-stack.md)
- [Self-Host AI Inside Office](https://www.gatellm.io/use-cases/self-hosted-office-ai.md)
- [Token Usage Visibility & Attribution](https://www.gatellm.io/use-cases/token-ops.md)
- [Token FinOps: AI Cost Allocation & Control](https://www.gatellm.io/use-cases/token-finops.md)
- [MCP Tool Governance](https://www.gatellm.io/use-cases/mcp-tool-governance.md)
- [Web Search for Every Model](https://www.gatellm.io/use-cases/web-search-grounding.md)
- [Multi-Key Rotation & Standby](https://www.gatellm.io/use-cases/multi-key-pooling.md)
- [Admission Control for Connected Clients](https://www.gatellm.io/use-cases/client-admission-control.md)
- [Enforce Which Model Agents Actually Run](https://www.gatellm.io/use-cases/enforce-model-policy.md)
- [No Model Choice on the Client](https://www.gatellm.io/use-cases/automatic-model-tiering.md)
- [One Protocol, Any Model](https://www.gatellm.io/use-cases/one-protocol-any-model.md)
- [Office / Microsoft 365 Add-in](https://www.gatellm.io/office-addin.md)
- [Glossary](https://www.gatellm.io/glossary.md)
- [Enterprise AI Governance](https://www.gatellm.io/enterprise-ai-governance.md)
- [Cost Comparison Methodology](https://www.gatellm.io/methodology.md)
- [Model Catalog](https://www.gatellm.io/models.md)
- [SDK & Framework Integrations](https://www.gatellm.io/integrations.md)
- [AI Coding Agents & Harnesses](https://www.gatellm.io/coding-agents.md)
- [Security & Compliance](https://www.gatellm.io/security.md)
- [Privacy Policy](https://www.gatellm.io/privacy-policy.md)
- [Terms of Service](https://www.gatellm.io/terms.md)
- [Cookie Policy](https://www.gatellm.io/cookies.md)
- [Data Processing Agreement (DPA)](https://www.gatellm.io/dpa.md)
- [Sub-processor List](https://www.gatellm.io/subprocessors.md)
- [Security Questionnaire (CAIQ/SIG/VSA)](https://www.gatellm.io/trust.md)
- [Incident Response & Support](https://www.gatellm.io/sla.md)
- [Vulnerability Disclosure Policy](https://www.gatellm.io/vulnerability-disclosure.md)
- [Open Source License Notice](https://www.gatellm.io/open-source-licenses.md)
- [Version Support & Lifecycle (EOL)](https://www.gatellm.io/lifecycle.md)
- [Engineering Blog](https://www.gatellm.io/blog.md)
- [Your Agents Have More Permissions Than Your Employees: Governing MCP Tools at the Gateway](https://www.gatellm.io/blog/mcp-tool-governance.md)
- [The $10k Invoice Problem: Five Ways to Cut LLM Spend Without Dumbing Down Your App](https://www.gatellm.io/blog/reduce-cost.md)
- [Your Vendor's List Price Is Not Your Cost: Applying Real FinOps to Token Spend](https://www.gatellm.io/blog/token-finops.md)
- [llms.txt](https://www.gatellm.io/llms.txt)