AI Gateway FAQ

Only if the gateway is licensed per instance rather than marking up tokens. GateLLM is licensed per 4GB instance (Pro $400/mo) — never a per-token markup, no platform or payment surcharge; you pay providers at standard rates. The real savings come from Composition: a top-tier model (Claude Opus) plans while open SOTA models (GLM / Qwen / DeepSeek) execute — same output at ~1/10 the cost. SaaS aggregators typically add ~5–8% platform + payment fees: a material hidden cost at volume.

Honest bar: single provider, single team, no compliance constraint, manageable monthly spend — calling APIs directly is simpler and you do not need a gateway. A gateway starts paying for itself when you need multi-provider failover, per-team cost attribution, compliance / data residency, or real-time visibility into runaway spend. Do not adopt one before you genuinely need it.

Direct-to-one-provider is a single point of failure — one outage takes your whole app down. GateLLM supports cross-provider fallback: when the primary returns 5xx or times out, traffic moves to a backup provider with no code changes. For more, use Ensemble: one logical call fans out to N models and is settled by vote / judge / best-of-N — reliability a single model cannot give you.

A SaaS gateway adds an extra hop out of your network. GateLLM is self-hosted inside your VPC; the extra hop to the vendor API is local inter-process communication — sub-millisecond overhead, with negligible impact on streaming time-to-first-token.

Yes. GateLLM Composition supports mode-aware policy: admins lock plan / execute tiers — top-tier closed models for planning, open SOTA for execution — for ~98% capability at ~10% cost. You can also classify by token count or complexity to route dynamically, preventing all-traffic-on-the-most-expensive-model.

A poor proxy does: it buffers the stream, breaks tool_calls JSON fragments, or trips on cross-vendor tool-calling format differences. GateLLM passes streams through natively without buffering and does capability normalization — unifying tool calling, structured output, prompt cache, reasoning tokens, and three other cross-vendor coupling points behind one interface, so composition / ensemble never breaks on format drift.

A relay forwards your requests through its own servers — your prompts, keys, and outputs are all exposed to it, with real risks of model substitution, degradation, or the operator running off (a CISPA study found 45.83% of relay endpoints could not pass model-fingerprint verification). GateLLM is BYOK and self-hosted: keys live only in gateway memory, requests route straight to vendor APIs, no third party in the path — there is no room to substitute, degrade, or disappear.

Yes. GateLLM deploys fully inside your network — data never leaves, satisfying data-export compliance. It natively speaks the DashScope protocol to reach domestic models (Tongyi / Zhipu / DeepSeek / Doubao), supports Xinchuang (Kunpeng / Hygon) and MLPS Level 3. Closed-model traffic is reduced via Composition, cutting cross-border calls.

Monthly invoices are too late. GateLLM Token Ops attributes cost per request — by team / agent / model / client — with real-time burn-rate visibility, and per-agent budget caps that actually halt calls. Crash-safe billing (pre-deduct → settle → refund) stays correct across process crashes; the Management API exports reconciliation data into your internal billing system.

With a SaaS aggregator you can only trust its word — there is no cryptographic verification that it is not quietly swapping in a smaller model to pocket the margin (OpenRouter itself does no model-identity verification). GateLLM is BYOK and self-hosted: you use your own keys and requests go from your gateway straight to the vendor API endpoint — model identity is guaranteed by the vendor API itself, with no replaceable hop in between.

Yes. Point your OpenAI SDK base_url at GateLLM (with /v1) — no other code changes — and call every model through one unified interface. Anthropic Messages, Gemini, and DashScope protocols are also supported as ingress — any-to-any bidirectional conversion. One base_url swap completes migration.

LiteLLM is an excellent open-source LLM proxy that translates the API envelope. GateLLM builds on top: capability normalization (unifying tool calling, structured output, and the other coupling points), Composition, Ensemble, and full self-hosted enterprise governance (SSO/SCIM, audit, MLPS / Xinchuang). See /compare/litellm.

Yes. Fully self-hosted in your VPC; BYOK keys in memory only, never on disk; SSO/SCIM (SAML / OIDC / Feishu / DingTalk); request-level audit logs, exportable; SOC 2 Type II / ISO 27001 / GDPR / MLPS Level 3 / Xinchuang ready. See /security.