AI Gateway FAQ

Only if the gateway is licensed per instance rather than marking up tokens. GateLLM is licensed per 2GB instance (Pro $400/mo) — never a per-token markup, no platform or payment surcharge; you pay providers at standard rates. The real savings come from Composition: a top-tier model (Claude Opus) plans while open SOTA models (GLM / Qwen / DeepSeek) execute — same output at ~1/10 the cost. SaaS aggregators typically add ~5–8% platform + payment fees: a material hidden cost at volume.

Honest bar: single provider, single team, no compliance constraint, manageable monthly spend — calling APIs directly is simpler and you do not need a gateway. A gateway starts paying for itself when you need multi-provider failover, per-team cost attribution, compliance / data residency, or real-time visibility into runaway spend. Do not adopt one before you genuinely need it.

Direct-to-one-provider is a single point of failure — one outage takes your whole app down. GateLLM supports cross-provider fallback: when the primary returns 5xx or times out, traffic moves to a backup provider with no code changes. For stronger reliability, run multi-model consensus in your orchestration layer (e.g. GitHub Actions parallel review) — the gateway supplies protocol interop so several models check the same request, settled by vote / judge.

A SaaS gateway adds an extra hop out of your network. GateLLM is self-hosted inside your VPC; the extra hop to the vendor API is local inter-process communication — sub-millisecond overhead, with negligible impact on streaming time-to-first-token.

Yes. GateLLM Composition supports mode-aware policy: admins lock plan / execute tiers — top-tier closed models for planning, open SOTA for execution — for ~98% capability at ~10% cost. You can also classify by token count or complexity to route dynamically, preventing all-traffic-on-the-most-expensive-model.

A poor proxy does: it buffers the stream, breaks tool_calls JSON fragments, or trips on cross-vendor tool-calling format differences. GateLLM passes streams through natively without buffering and does capability normalization — unifying tool calling, structured output, prompt cache, reasoning tokens, and three other cross-vendor coupling points behind one interface, so cross-vendor model mixing never breaks on format drift.

A relay forwards your requests through its own servers — your prompts, keys, and outputs are all exposed to it, with real risks of model substitution, degradation, or the operator running off (a CISPA study found identity-verification failures in 45.83% of model-fingerprint tests, arxiv.org/abs/2603.01919). GateLLM is BYOK and self-hosted: keys live only in gateway memory, requests route straight to vendor APIs, no third party in the path — there is no room to substitute, degrade, or disappear.

Yes. GateLLM deploys fully inside your network — data never leaves, satisfying data-export compliance. It natively speaks the DashScope protocol to reach domestic models (Tongyi / Zhipu / DeepSeek / Doubao). Closed-model traffic is reduced via Composition, cutting cross-border calls.

Monthly invoices are too late. GateLLM Token Ops attributes cost per request — by team / agent / model / client — with real-time burn-rate visibility, and per-agent budget caps that actually halt calls. Crash-safe billing (pre-deduct → settle → refund) stays correct across process crashes; the Management API exports reconciliation data into your internal billing system.

With a SaaS aggregator you can only trust its word — there is no cryptographic verification that it is not quietly swapping in a smaller model to pocket the margin (OpenRouter itself does no model-identity verification). GateLLM is BYOK and self-hosted: you use your own keys and requests go from your gateway straight to the vendor API endpoint — model identity is guaranteed by the vendor API itself, with no replaceable hop in between.

Yes. Point your OpenAI SDK base_url at GateLLM (with /v1) — no other code changes — and call every model through one unified interface. One base_url swap completes migration: no feature drift, no proxy shim — your OpenAI client drives a vendor-equivalent unified endpoint directly.

No — one gateway serves all three. Point the OpenAI SDK base_url at the gateway (with /v1); point the Anthropic SDK at the gateway root (the SDK appends /v1/messages itself, key via the x-api-key header); the Google Gemini SDK connects directly (key via the x-goog-api-key header). The gateway translates bidirectionally across four ingress + four egress protocols (OpenAI / Anthropic / Gemini / DashScope) — client and upstream protocols may differ: call Claude from the OpenAI SDK, GPT from the Anthropic SDK, or DeepSeek from the Gemini SDK with zero code changes; third-party clients like Cherry Studio use the OpenAI-compatible ingress. See the protocol interop matrix at docs.gatellm.io reference/protocol-matrix.

Yes. Bedrock connects as an upstream (aws_converse / aws_invoke protocol, Bedrock API key + regional runtime endpoint, inference profiles supported); clients keep using the OpenAI / Anthropic / Gemini SDKs and the gateway translates the protocol. Bedrock-hosted Claude / GLM and direct-vendor models can mix in the same routing / load balancer under one governance plane.

Yes. The gateway has a built-in MCP proxy: it aggregates multiple external MCP tool servers (streamable HTTP / SSE) behind one /mcp endpoint, so Claude Desktop or any MCP client calls them all with a single gateway key. Per-key-group tool ACLs and context budgets govern access, with built-in BM25 search when the tool list gets long. Tool admission, rate limiting, and audit share the same governance plane as model traffic.

Yes. The gateway bridges WebSocket sessions at /v1/realtime (OpenAI Realtime and DashScope Realtime protocols). Long-lived voice sessions run in parallel with HTTP traffic: same key authentication and admission, metered per turn (response.create → response.done) into the same statistics / billing / log pipeline as HTTP — one consistent cost ledger.

Five layers: key groups scope which models and load balancers each tenant can reach; access keys + Header ACL rules differentiate clients by request header; identity-scoped model mapping routes the same model name to different upstreams per tenant; per-key RPM / concurrency / TPM limits cap usage; and per-request cost attribution means each tenant sees only its own spend. See the multi-tenant isolation guide at docs.gatellm.io usecases/multi-tenant-isolation.

Yes. GateLLM hijacks search at the gateway layer: configure a search engine (serper, testable inline in the console), then per-model three-way toggles (follow capability / force on / force off) control injection. Results are synthesized into native search blocks per client protocol (Anthropic server_tool_use / OpenAI url_citation / Responses web_search_call) — open models get search grounding too. Search failures degrade gracefully without blocking requests, and hijacked search is excluded from usage billing. See console/search at docs.gatellm.io.

Yes. One upstream can hold multiple API keys with weighted distribution: a deterministic weighted hash (FNV-1a) stickily binds each caller to the same key (cache affinity), and when a key fails the gateway retries with the next untried key inside that upstream. Keys with weight=0 are standby — used only after every weight>0 key has failed. Binding state is observable row by row in the console. See reference/upstreams-models-fields at docs.gatellm.io.

Yes. Any OpenAI-compatible endpoint works as an upstream: set a custom base URL and vLLM / Ollama / TGI services are governed alongside cloud vendors. Outbound proxies (socks5 / http), custom request headers, and per-upstream DNS are all configurable for private networks and special topologies.

No YAML required. GateLLM ships a visual console: upstreams / models / load balancers / access control / MCP / statistics / logs are all GUI-configured, with instant Chinese/English switching. Batch changes go through stage → preview (full namespace-conflict set) → atomic commit, rolling back the whole batch on any conflict. Console accounts come in admin / monitor (read-only) roles, and every operation and login is audited.

LiteLLM is an excellent open-source LLM proxy. GateLLM and LiteLLM share a capability baseline (routing, tool calling, structured output, caching, SSO via OIDC + SAML 2.0 + SCIM). GateLLM's differentiators are the single Rust binary, 15 protocol shapes (the 4 chat-protocol families translate any-to-any, the rest per interop matrix), gateway-side cache auto-injection, and a YAML-free console; LiteLLM leads on its enterprise ecosystem (secret managers, guardrails, observability callbacks). See /compare/litellm.

Yes. Fully self-hosted in your VPC; BYOK keys in memory only, never on disk; enterprise SSO (OIDC: Azure AD / Okta / generic, plus SAML 2.0 and SCIM 2.0 auto-provisioning); request-level audit logs, exportable; GDPR ready; SOC 2 Type II / ISO 27001 controls aligned (certification in preparation). See /security.

In Console → Settings → Billing, create a manual price snapshot for the model — it is marked pinned (yellow badge). A pinned price becomes current and shadows the auto baseline beneath it; system refreshes skip that model and never overwrite your manual price. Deleting it falls back to the next auto row and self-heals.

No. Price snapshots are ordered by created_at and each covers only its [own time, next time) window — the earliest backfills history, the latest extends forward indefinitely. Create a new snapshot with a future effective time to pre-schedule the cut so it only affects later months, or backfill a historical price to cost past months precisely.

Set a daily / monthly cost quota (USD amount) on the access key — two independent cycles judged on UTC calendar day and month, returning 429 quota_exceeded and stopping service the moment either trips (the daily window carries Retry-After). Quotas inherit key → group → global; a group quota is a per-key cap within the group (not a shared ledger), and across groups the widest wins.