# AI Gateway FAQ - GateLLM > AI gateway FAQ: Is it cheaper than calling OpenAI directly? Do I need one? How does it handle provider outages? How is it different from a relay/aggregator? Can I use it in China? Answered by GateLLM, a self-hosted BYOK gateway. Canonical: https://www.gatellm.io/faq/ ## AI Gateway FAQ Real questions on AI gateway / LLM gateway / LLM proxy selection, cost, reliability, and compliance — answered. These questions are drawn from what developers and architects actually ask about AI gateways on Reddit, StackOverflow, and Zhihu — cost, failover, latency, relay risk, China compliance, tool-calling compatibility, multi-SDK access, MCP tools, Bedrock upstreams, realtime voice, multi-tenant isolation. GateLLM is a self-hosted, BYOK enterprise multi-model gateway: keys live in memory, requests go straight to vendor APIs, licensed per instance with no token markup. ### Is it cheaper to use an AI gateway than calling OpenAI or Anthropic directly? Only if the gateway is licensed per instance rather than marking up tokens. GateLLM is licensed per 2GB instance (Pro $400/mo) — never a per-token markup, no platform or payment surcharge; you pay providers at standard rates. The real savings come from Composition: a top-tier model (Claude Opus) plans while open SOTA models (GLM / Qwen / DeepSeek) execute — at ~1/7 the cost. SaaS aggregators typically add ~5–8% platform + payment fees: a material hidden cost at volume. ### Do I need an AI gateway, or should I just call provider APIs directly? Honest bar: single provider, single team, no compliance constraint, manageable monthly spend — calling APIs directly is simpler and you do not need a gateway. A gateway starts paying for itself when you need multi-provider failover, per-team cost attribution, compliance / data residency, or real-time visibility into runaway spend. Do not adopt one before you genuinely need it. ### How does an AI gateway handle provider outages (e.g. OpenAI 503)? Direct-to-one-provider is a single point of failure — one outage takes your whole app down. GateLLM supports cross-provider fallback: when the primary returns 5xx or times out, traffic moves to a backup provider with no code changes. Separately, for higher-stakes calls you can run a multi-model code review in your orchestration layer (e.g. GitHub Actions parallel review) — the gateway supplies protocol interop so several models check the same request. ### How much latency does an AI gateway add? A SaaS gateway adds an extra hop out of your network. GateLLM is self-hosted inside your VPC; the extra hop to the vendor API is local inter-process communication — sub-millisecond overhead, with negligible impact on streaming time-to-first-token. ### Can the gateway route simple tasks to cheaper models automatically? Yes. GateLLM Composition supports mode-aware policy: admins lock plan / execute tiers — top-tier closed models for planning, open SOTA for execution — for ~97% capability at ~14% cost. You can also classify by token count or complexity to route dynamically, preventing all-traffic-on-the-most-expensive-model. ### Does an AI gateway break streaming (SSE) or tool calling? A poor proxy does: it buffers the stream, breaks tool_calls JSON fragments, or trips on cross-vendor tool-calling format differences. GateLLM passes streams through natively without buffering and does capability normalization — unifying tool calling, structured output, prompt cache, reasoning tokens, and three other cross-vendor coupling points behind one interface, so cross-vendor model mixing never breaks on format drift. ### How is GateLLM different from an AI relay / proxy service? A relay forwards your requests through its own servers — your prompts, keys, and outputs are all exposed to it, with real risks of model substitution, degradation, or the operator running off (a CISPA study found identity-verification failures in 45.83% of model-fingerprint tests, arxiv.org/abs/2603.01919). GateLLM is BYOK and self-hosted: keys live only in gateway memory, requests route straight to vendor APIs, no third party in the path — there is no room to substitute, degrade, or disappear. ### Can I use GateLLM in China with domestic models and data-residency requirements? Yes. GateLLM deploys fully inside your network — data never leaves, satisfying data-export compliance. It natively speaks the DashScope protocol to reach domestic models (Tongyi / Zhipu / DeepSeek / Doubao). Closed-model traffic is reduced via Composition, cutting cross-border calls. ### How do I track AI cost burn rate and reconcile with provider bills? Monthly invoices are too late. GateLLM Token Ops attributes cost per request — by team / agent / model / client — with real-time burn-rate visibility, and per-agent budget caps that actually halt calls. Crash-safe billing (pre-deduct → settle → refund) stays correct across process crashes; the Management API exports reconciliation data into your internal billing system. ### How do I know the gateway is serving the real model, not a cheaper substitute? With a SaaS aggregator you can only trust its word — there is no cryptographic verification that it is not quietly swapping in a smaller model to pocket the margin (OpenRouter itself does no model-identity verification). GateLLM is BYOK and self-hosted: you use your own keys and requests go from your gateway straight to the vendor API endpoint — model identity is guaranteed by the vendor API itself, with no replaceable hop in between. ### Does GateLLM work with the OpenAI SDK? How do I migrate? Yes. Point your OpenAI SDK base_url at GateLLM (with /v1) — no other code changes — and call every model through one unified interface. One base_url swap completes migration: no feature drift, no proxy shim — your OpenAI client drives a vendor-equivalent unified endpoint directly. ### My team uses the OpenAI, Anthropic, and Google SDKs — do I need separate gateways? No — one gateway serves all three. Point the OpenAI SDK base_url at the gateway (with /v1); point the Anthropic SDK at the gateway root (the SDK appends /v1/messages itself, key via the x-api-key header); the Google Gemini SDK connects directly (key via the x-goog-api-key header). The gateway translates bidirectionally across four ingress + four egress protocols (OpenAI / Anthropic / Gemini / DashScope) — client and upstream protocols may differ: call Claude from the OpenAI SDK, GPT from the Anthropic SDK, or DeepSeek from the Gemini SDK with zero code changes; third-party clients like Cherry Studio use the OpenAI-compatible ingress. See the protocol interop matrix at docs.gatellm.io reference/protocol-matrix. ### Can GateLLM front models hosted on AWS Bedrock? Yes. Bedrock connects as an upstream (aws_converse / aws_invoke protocol, Bedrock API key + regional runtime endpoint, inference profiles supported); clients keep using the OpenAI / Anthropic / Gemini SDKs and the gateway translates the protocol. Bedrock-hosted Claude / GLM and direct-vendor models can mix in the same routing / load balancer under one governance plane. ### Does GateLLM support MCP (Model Context Protocol) tools? Yes. The gateway has a built-in MCP proxy: it aggregates multiple external MCP tool servers (streamable HTTP / SSE) behind one /mcp endpoint, so Claude Desktop or any MCP client calls them all with a single gateway key. Per-key-group tool ACLs and context budgets govern access, with built-in BM25 search when the tool list gets long. Tool admission, rate limiting, and audit share the same governance plane as model traffic. ### Does it support realtime voice sessions (Realtime API)? Yes. The gateway bridges WebSocket sessions at /v1/realtime (OpenAI Realtime and DashScope Realtime protocols). Long-lived voice sessions run in parallel with HTTP traffic: same key authentication and admission, metered per turn (response.create → response.done) into the same statistics / billing / log pipeline as HTTP — one consistent cost ledger. ### How do I isolate multiple teams / tenants on one gateway? Five layers: key groups scope which models and load balancers each tenant can reach; access keys + Header ACL rules differentiate clients by request header; identity-scoped model mapping routes the same model name to different upstreams per tenant; per-key RPM / concurrency / TPM limits cap usage; and per-request cost attribution means each tenant sees only its own spend. See the multi-tenant isolation guide at docs.gatellm.io usecases/multi-tenant-isolation. ### Open models have no web search — can the gateway add it uniformly? Yes. GateLLM hijacks search at the gateway layer: configure a search engine (serper, testable inline in the console), then per-model three-way toggles (follow capability / force on / force off) control injection. Results are synthesized into native search blocks per client protocol (Anthropic server_tool_use / OpenAI url_citation / Responses web_search_call) — open models get search grounding too. Search failures degrade gracefully without blocking requests, and hijacked search is excluded from usage billing. See console/search at docs.gatellm.io. ### I have multiple API keys for one provider — can the gateway rotate them? What when a key gets rate-limited? Yes. One upstream can hold multiple API keys with weighted distribution: a deterministic weighted hash (FNV-1a) stickily binds each caller to the same key (cache affinity), and when a key fails the gateway retries with the next untried key inside that upstream. Keys with weight=0 are standby — used only after every weight>0 key has failed. Binding state is observable row by row in the console. See reference/upstreams-models-fields at docs.gatellm.io. ### Can self-hosted open models (vLLM / Ollama) plug into the gateway? Yes. Any OpenAI-compatible endpoint works as an upstream: set a custom base URL and vLLM / Ollama / TGI services are governed alongside cloud vendors. Outbound proxies (socks5 / http), custom request headers, and per-upstream DNS are all configurable for private networks and special topologies. ### Do I have to write YAML? Is there a visual console? No YAML required. GateLLM ships a visual console: upstreams / models / load balancers / access control / MCP / statistics / logs are all GUI-configured, with instant Chinese/English switching. Batch changes go through stage → preview (full namespace-conflict set) → atomic commit, rolling back the whole batch on any conflict. Console accounts come in admin / monitor (read-only) roles, and every operation and login is audited. ### How is GateLLM different from LiteLLM? LiteLLM is an excellent open-source LLM proxy. GateLLM and LiteLLM share a capability baseline (routing, tool calling, structured output, caching, SSO via OIDC + SAML 2.0 + SCIM). GateLLM's differentiators are the single Rust binary, 15 protocol shapes (the 4 chat-protocol families translate any-to-any, the rest per interop matrix), gateway-side cache auto-injection, and a YAML-free console; LiteLLM leads on its enterprise ecosystem (secret managers, third-party guardrail callbacks, observability callbacks). See /compare/litellm. ### Does GateLLM support VPC deployment, SSO, and audit logging? Yes. Fully self-hosted in your VPC; BYOK keys in memory only, never on disk; enterprise SSO (OIDC: Azure AD / Okta / generic, plus SAML 2.0 and SCIM 2.0 auto-provisioning); request-level audit logs, exportable; GDPR ready; SOC 2 Type II / ISO 27001 controls aligned (certification in preparation). See /security. ### How do I record the private price I negotiated with a vendor? In Console → Settings → Billing, create a manual price snapshot for the model — it is marked pinned (yellow badge). A pinned price becomes current and shadows the auto baseline beneath it; system refreshes skip that model and never overwrite your manual price. Deleting it falls back to the next auto row and self-heals. ### If a model drops its price mid-cycle, do historical months get polluted? No. Price snapshots are ordered by created_at and each covers only its [own time, next time) window — the earliest backfills history, the latest extends forward indefinitely. Create a new snapshot with a future effective time to pre-schedule the cut so it only affects later months, or backfill a historical price to cost past months precisely. ### How do I cap a key or team's spend per day or per month? Set a daily / monthly cost quota (USD amount) on the access key — two independent cycles judged on UTC calendar day and month, returning 429 quota_exceeded and stopping service the moment either trips (the daily window carries Retry-After). Quotas inherit key → group → global; a group quota is a per-key cap within the group (not a shared ledger), and across groups the widest wins. Didn't find your question? Talk to us. ## Related - [Home](https://www.gatellm.io/index.md) - [Pricing](https://www.gatellm.io/pricing.md) - [Cost Calculator](https://www.gatellm.io/calculator.md) - [About](https://www.gatellm.io/about.md) - [Book a Demo](https://www.gatellm.io/demo.md) - [Compare AI Gateways](https://www.gatellm.io/compare.md) - [Compare LiteLLM](https://www.gatellm.io/compare/litellm.md) - [Compare OpenRouter](https://www.gatellm.io/compare/openrouter.md) - [Compare Alibaba Cloud AI Gateway](https://www.gatellm.io/compare/alibaba-cloud.md) - [Compare Azure API Management](https://www.gatellm.io/compare/azure-apim.md) - [Compare Apigee](https://www.gatellm.io/compare/apigee.md) - [Compare Cloudflare AI Gateway](https://www.gatellm.io/compare/cloudflare.md) - [Compare Kong AI Gateway](https://www.gatellm.io/compare/kong.md) - [Compare Portkey](https://www.gatellm.io/compare/portkey.md) - [LiteLLM vs Kong AI Gateway](https://www.gatellm.io/compare/litellm-vs-kong.md) - [LiteLLM vs OpenRouter](https://www.gatellm.io/compare/litellm-vs-openrouter.md) - [LiteLLM vs Portkey](https://www.gatellm.io/compare/litellm-vs-portkey.md) - [Use Cases](https://www.gatellm.io/use-cases.md) - [Reduce LLM Cost](https://www.gatellm.io/use-cases/reduce-cost.md) - [High Availability](https://www.gatellm.io/use-cases/high-availability.md) - [Multi-Tenant Isolation](https://www.gatellm.io/use-cases/multi-tenant.md) - [Compliance & Audit](https://www.gatellm.io/use-cases/compliance-audit.md) - [Troubleshooting by Symptom](https://www.gatellm.io/use-cases/troubleshooting.md) - [Mixed Multi-Vendor Model Stack](https://www.gatellm.io/use-cases/mixed-model-stack.md) - [Self-Host AI Inside Office](https://www.gatellm.io/use-cases/self-hosted-office-ai.md) - [Token Usage Visibility & Attribution](https://www.gatellm.io/use-cases/token-ops.md) - [Token FinOps: AI Cost Allocation & Control](https://www.gatellm.io/use-cases/token-finops.md) - [MCP Tool Governance](https://www.gatellm.io/use-cases/mcp-tool-governance.md) - [Web Search for Every Model](https://www.gatellm.io/use-cases/web-search-grounding.md) - [Multi-Key Rotation & Standby](https://www.gatellm.io/use-cases/multi-key-pooling.md) - [Admission Control for Connected Clients](https://www.gatellm.io/use-cases/client-admission-control.md) - [Enforce Which Model Agents Actually Run](https://www.gatellm.io/use-cases/enforce-model-policy.md) - [No Model Choice on the Client](https://www.gatellm.io/use-cases/automatic-model-tiering.md) - [One Protocol, Any Model](https://www.gatellm.io/use-cases/one-protocol-any-model.md) - [Office / Microsoft 365 Add-in](https://www.gatellm.io/office-addin.md) - [Glossary](https://www.gatellm.io/glossary.md) - [Model Catalog](https://www.gatellm.io/models.md) - [SDK & Framework Integrations](https://www.gatellm.io/integrations.md) - [AI Coding Agents & Harnesses](https://www.gatellm.io/coding-agents.md) - [Security & Compliance](https://www.gatellm.io/security.md) - [Privacy Policy](https://www.gatellm.io/privacy-policy.md) - [Terms of Service](https://www.gatellm.io/terms.md) - [Cookie Policy](https://www.gatellm.io/cookies.md) - [Data Processing Agreement (DPA)](https://www.gatellm.io/dpa.md) - [Sub-processor List](https://www.gatellm.io/subprocessors.md) - [Security Questionnaire (CAIQ/SIG/VSA)](https://www.gatellm.io/trust.md) - [Incident Response & Support](https://www.gatellm.io/sla.md) - [Vulnerability Disclosure Policy](https://www.gatellm.io/vulnerability-disclosure.md) - [Open Source License Notice](https://www.gatellm.io/open-source-licenses.md) - [Version Support & Lifecycle (EOL)](https://www.gatellm.io/lifecycle.md) - [Engineering Blog](https://www.gatellm.io/blog.md) - [Your Agents Have More Permissions Than Your Employees: Governing MCP Tools at the Gateway](https://www.gatellm.io/blog/mcp-tool-governance.md) - [The $10k Invoice Problem: Five Ways to Cut LLM Spend Without Dumbing Down Your App](https://www.gatellm.io/blog/reduce-cost.md) - [Your Vendor's List Price Is Not Your Cost: Applying Real FinOps to Token Spend](https://www.gatellm.io/blog/token-finops.md) - [llms.txt](https://www.gatellm.io/llms.txt)