# Multiple keys per provider: weighted distribution, sticky binding, standby takeover > AI gateway with multiple API keys per upstream: deterministic weighted hash, sticky per-caller binding, weight=0 standby takeover, observable bindings — stacked orthogonally with cross-upstream load balancing. Canonical: https://www.gatellm.io/use-cases/multi-key-pooling/ Attach multiple API keys to one upstream: weighted distribution, sticky per-caller binding, weight=0 standby keys as the backstop — no more hard waits on 429. ## What the community actually complains about Reddit favorites: "one OpenAI key's RPM is not enough, how do I rotate across several accounts", "a key hits 429 and the business just waits", "switching keys loses all session cache affinity". Rolling your own rotation means handling stickiness, weights, standby, and failure eviction yourself — real engineering effort. ## How GateLLM does it The GateLLM upstream form accepts multiple API keys: a deterministic weighted hash (FNV-1a) stickily binds each caller to the same upstream key (stable within the TTL; rotating a key in place does not break stickiness), and weights set the distribution ratio. When a key fails, the gateway retries with the next untried key inside that upstream — without overriding the original sticky binding. Keys with weight=0 are standby, enabled only after every weight>0 key has failed. Binding state (binding / predicted / stale) is observable row by row in the console. The mechanism stacks orthogonally with load balancers: Smooth WRR picks the upstream across entries, then sticky selection picks the key inside it. ## 5 levers that actually land - Weighted multi-key: deterministic weighted hash distributes callers, weights are ratios - Sticky binding: each caller stays on the same upstream key within the TTL — cache affinity - Standby keys: weight=0 keys sit out normal distribution, enabled only after every active key fails - Observable bindings: binding / predicted / stale states row by row, auto-refreshing every 3s - Orthogonal to LB: key selection inside an upstream + Smooth WRR across upstreams stack as two layers ## FAQ ### Does key rotation lose prompt-cache affinity? Sticky binding exists exactly for that: each caller stays bound to the same upstream key within the TTL, hitting the same vendor account's cache. Rotating a key in place does not break stickiness; only adding / removing / reordering keys or changing weights triggers reselection. ### When does a standby key take effect? weight=0 keys stay out of normal distribution and are enabled only after every weight>0 key has been tried and failed — and they hold no binding. When a key fails mid-flight, the gateway retries with the next untried key inside the same upstream without overriding the original sticky binding: that caller's later requests still start from the originally bound key. - [Full reference: Upstream & model fields (docs)](https://docs.gatellm.io/en/reference/upstreams-models-fields) ## Related - [Home](https://www.gatellm.io/index.md) - [Pricing](https://www.gatellm.io/pricing.md) - [Cost Calculator](https://www.gatellm.io/calculator.md) - [About](https://www.gatellm.io/about.md) - [Book a Demo](https://www.gatellm.io/demo.md) - [FAQ](https://www.gatellm.io/faq.md) - [Compare AI Gateways](https://www.gatellm.io/compare.md) - [Compare LiteLLM](https://www.gatellm.io/compare/litellm.md) - [Compare OpenRouter](https://www.gatellm.io/compare/openrouter.md) - [Compare Alibaba Cloud AI Gateway](https://www.gatellm.io/compare/alibaba-cloud.md) - [Compare Azure API Management](https://www.gatellm.io/compare/azure-apim.md) - [Compare Apigee](https://www.gatellm.io/compare/apigee.md) - [Compare Cloudflare AI Gateway](https://www.gatellm.io/compare/cloudflare.md) - [Compare Kong AI Gateway](https://www.gatellm.io/compare/kong.md) - [Compare Portkey](https://www.gatellm.io/compare/portkey.md) - [LiteLLM vs Kong AI Gateway](https://www.gatellm.io/compare/litellm-vs-kong.md) - [LiteLLM vs OpenRouter](https://www.gatellm.io/compare/litellm-vs-openrouter.md) - [LiteLLM vs Portkey](https://www.gatellm.io/compare/litellm-vs-portkey.md) - [Use Cases](https://www.gatellm.io/use-cases.md) - [Reduce LLM Cost](https://www.gatellm.io/use-cases/reduce-cost.md) - [High Availability](https://www.gatellm.io/use-cases/high-availability.md) - [Multi-Tenant Isolation](https://www.gatellm.io/use-cases/multi-tenant.md) - [Compliance & Audit](https://www.gatellm.io/use-cases/compliance-audit.md) - [Troubleshooting by Symptom](https://www.gatellm.io/use-cases/troubleshooting.md) - [Mixed Multi-Vendor Model Stack](https://www.gatellm.io/use-cases/mixed-model-stack.md) - [Self-Host AI Inside Office](https://www.gatellm.io/use-cases/self-hosted-office-ai.md) - [Token Usage Visibility & Attribution](https://www.gatellm.io/use-cases/token-ops.md) - [Token FinOps: AI Cost Allocation & Control](https://www.gatellm.io/use-cases/token-finops.md) - [MCP Tool Governance](https://www.gatellm.io/use-cases/mcp-tool-governance.md) - [Web Search for Every Model](https://www.gatellm.io/use-cases/web-search-grounding.md) - [Admission Control for Connected Clients](https://www.gatellm.io/use-cases/client-admission-control.md) - [Enforce Which Model Agents Actually Run](https://www.gatellm.io/use-cases/enforce-model-policy.md) - [No Model Choice on the Client](https://www.gatellm.io/use-cases/automatic-model-tiering.md) - [One Protocol, Any Model](https://www.gatellm.io/use-cases/one-protocol-any-model.md) - [Office / Microsoft 365 Add-in](https://www.gatellm.io/office-addin.md) - [Glossary](https://www.gatellm.io/glossary.md) - [Model Catalog](https://www.gatellm.io/models.md) - [SDK & Framework Integrations](https://www.gatellm.io/integrations.md) - [AI Coding Agents & Harnesses](https://www.gatellm.io/coding-agents.md) - [Security & Compliance](https://www.gatellm.io/security.md) - [Privacy Policy](https://www.gatellm.io/privacy-policy.md) - [Terms of Service](https://www.gatellm.io/terms.md) - [Cookie Policy](https://www.gatellm.io/cookies.md) - [Data Processing Agreement (DPA)](https://www.gatellm.io/dpa.md) - [Sub-processor List](https://www.gatellm.io/subprocessors.md) - [Security Questionnaire (CAIQ/SIG/VSA)](https://www.gatellm.io/trust.md) - [Incident Response & Support](https://www.gatellm.io/sla.md) - [Vulnerability Disclosure Policy](https://www.gatellm.io/vulnerability-disclosure.md) - [Open Source License Notice](https://www.gatellm.io/open-source-licenses.md) - [Version Support & Lifecycle (EOL)](https://www.gatellm.io/lifecycle.md) - [Engineering Blog](https://www.gatellm.io/blog.md) - [Your Agents Have More Permissions Than Your Employees: Governing MCP Tools at the Gateway](https://www.gatellm.io/blog/mcp-tool-governance.md) - [The $10k Invoice Problem: Five Ways to Cut LLM Spend Without Dumbing Down Your App](https://www.gatellm.io/blog/reduce-cost.md) - [Your Vendor's List Price Is Not Your Cost: Applying Real FinOps to Token Spend](https://www.gatellm.io/blog/token-finops.md) - [llms.txt](https://www.gatellm.io/llms.txt)