The $10k Invoice Problem: Five Ways to Cut LLM Spend Without Dumbing Down Your App

There is a genre of post that shows up on r/LocalLLaMA every few weeks. The details vary, but the plot is always the same: "The monthly invoice arrived, and only then did I find out we burned $10k." Sometimes it is a runaway agent loop. Sometimes a retry storm. Sometimes it is simply growth.

The other recurring one: "60% of our requests could have run on a cheaper model, but everything goes to the most expensive one because that is what the code says."

Both are the same failure, and it is not a pricing failure. It is an observability and control failure. The invoice is a lagging indicator. By the time finance forwards it to you, the money is gone, and per-key rate limits — the only control most provider dashboards offer — are useless for business decisions.

If you are a single team calling one provider at modest volume, you do not need any of this: call the API directly and move on. If any of the following sounds familiar, the rest of this post is a playbook.

Lever 1: Know what you pay on top of the model price

Before optimizing anything, check the markup.

SaaS aggregators and relay services commonly add a platform and payment fee on top of the provider's own rate. That is not unreasonable — it is a business model — but it is worth naming: at high inference volume it becomes a recurring tax that buys you nothing you could not run yourself. The alternative shape is a flat per-instance license: you pay for the gateway software, providers bill you at their standard rates, and doubling token volume does not move the software fee. (Disclosure: the second shape is the one GateLLM uses — $400/mo for a 2GB Pro instance, no per-token markup.) The point stands regardless of vendor: know which side of the per-token line your gateway sits on.

Lever 2: Tiered routing — the lever that actually changes the number

The single largest cost reduction available to most teams is not a discount. It is running the right model per request instead of the same model for every request.

Most production traffic is not uniformly hard. Classification, extraction, summarization, formatting, and tool-argument construction rarely need a frontier model. Planning, ambiguous reasoning, and long-horizon agent steps often do. A tiered policy — top-tier closed model for planning, open SOTA model for execution — typically moves the blended cost per task by a large factor, because the cheap tier absorbs the volume.

Two implementation notes that decide whether this works:

• The routing decision must be per request, not per deployment. A single global "use the cheap model" switch will eventually break the hard 5% of traffic and get reverted. Decide by request shape, by plan mode, or by identity.
• Keep it invisible to clients. If tiering requires application changes, it will not survive the next refactor. Put the policy in the gateway so the client keeps sending the same request.

Lever 3: Make prompt caching visible, and audit its billing

Prompt caching is the cheapest win available, and the easiest to silently lose. Three failure modes show up repeatedly:

1. The cache breakpoint is never set, so nothing is ever cached — often because the client speaks a protocol that does not expose caching, or nobody added the marker.
2. The cache is set but not reported, so read tokens are billed at full input price with no visibility.
3. Cached-token accounting is wrong at the aggregation layer, inflating the reported bill against what the provider actually charged.

The fix is to make cache reads and writes first-class in your metering: same units, same attribution, visible per key and per team. If you cannot see cache hit rate next to cost, you cannot tell whether the optimization is working.

Lever 4: Trim the response, not the model

A surprising share of spend is output tokens nobody reads: boilerplate preambles, repeated context echoed back, verbose chain-of-thought that the caller discards. Output tokens are the expensive side of the price sheet, so trimming them is often cheaper than downgrading the model — and it does not cost you quality on the part of the answer a human actually sees.

Start by measuring output-to-input ratio per endpoint. Endpoints with unusually high ratios are usually the ones where a prompt or response-schema fix pays for itself immediately.

Lever 5: Attribution and hard stops, before the invoice

Cost control that lives in a spreadsheet is a report, not a control. The minimum viable version has three parts:

• Attribution: every request carries a key, team, or project, so cost lands on a line item rather than a single shared pool.
• Budgets with teeth: thresholds that can degrade or block traffic, not just alert on it.
• Real-time visibility: cost available while the request is running, not at month end.

The distinction between alerting and blocking is the whole point. A runaway agent loop is not dissuaded by a dashboard.

Where to start

If you do one thing this week, measure the ratio of spend to value per endpoint: which endpoints, teams, and keys consume the budget, and what fraction of their traffic actually needed the model they used. That number usually identifies the first lever on its own.

The broader pattern behind all five levers is the same: cost becomes controllable at the moment it becomes attributable and enforceable in real time. Everything else is accounting.