Multiple keys per provider: weighted distribution, sticky binding, standby takeover
Attach multiple API keys to one upstream: weighted distribution, sticky per-caller binding, weight=0 standby keys as the backstop — no more hard waits on 429.
What the community actually complains about
Reddit favorites: "one OpenAI key's RPM is not enough, how do I rotate across several accounts", "a key hits 429 and the business just waits", "switching keys loses all session cache affinity".
Rolling your own rotation means handling stickiness, weights, standby, and failure eviction yourself — real engineering effort.
How GateLLM does it
The GateLLM upstream form accepts multiple API keys: a deterministic weighted hash (FNV-1a) stickily binds each caller to the same upstream key (stable within the TTL; rotating a key in place does not break stickiness), and weights set the distribution ratio. When a key fails, the gateway retries with the next untried key inside that upstream — without overriding the original sticky binding.
Keys with weight=0 are standby, enabled only after every weight>0 key has failed. Binding state (binding / predicted / stale) is observable row by row in the console.
The mechanism stacks orthogonally with load balancers: Smooth WRR picks the upstream across entries, then sticky selection picks the key inside it.
5 levers that actually land
- Weighted multi-key: deterministic weighted hash distributes callers, weights are ratios
- Sticky binding: each caller stays on the same upstream key within the TTL — cache affinity
- Standby keys: weight=0 keys sit out normal distribution, enabled only after every active key fails
- Observable bindings: binding / predicted / stale states row by row, auto-refreshing every 3s
- Orthogonal to LB: key selection inside an upstream + Smooth WRR across upstreams stack as two layers