Make LLMs cheaper in production.
Production LLM cost optimization means lowering what an application spends on large language model calls without lowering output quality. The levers I lean on are prompt and response caching, structured outputs, provider failover, token reduction, and evals to prove quality held. This is the running guide to each.
Where the money leaks.
Prompt & response caching
Most production spend is the same context sent over and over: system prompts, tool definitions, retrieved documents. Cache the stable prefix and you pay full price once, then a fraction on every reuse. The real work is deciding what is actually stable and ordering prompts so the cacheable part comes first.
Structured outputs
Free-form text you then parse is where reliability and cost both leak. Constrain the model to a schema and you drop the retries caused by malformed responses, return only the fields you need, and simplify the code downstream. Cheaper and more reliable tend to move together here.
Provider failover
One provider is one point of failure. When an API degrades or rate-limits, requests should fail over to another (Anthropic to OpenAI, or the reverse) behind a single interface. Done well, the same setup lets you route each task to the cheapest model that clears your quality bar.
Token reduction
Every token in and out is billed. Trimming dead context, compressing history, and summarizing instead of replaying lowers the cost of every single call. The trap is trimming so hard that quality slips, which is why this only works next to measurement.
Evals
You cannot cut cost safely without knowing whether quality held. Evals turn "it feels fine" into a number, so you can drop to a cheaper model, prune a prompt, or switch providers and prove the output did not get worse. They are the safety net under every other lever here.
Worked examples, with the data.
Questions I get asked.
- What is production LLM cost optimization?
- It is the practice of lowering what an application spends on large language model calls without lowering output quality. The main levers are prompt and response caching, structured outputs, provider failover, token reduction, and evaluations that confirm quality held.
- How much can prompt caching reduce LLM costs?
- It depends on how much of each request is repeated, stable context. The more your calls reuse the same system prompt, tool definitions, and retrieved documents, the more caching helps, because you pay full price for that prefix once and a fraction on every reuse. Measure your own cache-hit rate rather than trusting a headline number.
- Do structured outputs make LLM calls cheaper?
- Often yes, in two ways. Constraining the model to a schema removes retries caused by malformed responses, and it lets you return only the fields you need instead of long prose. Both cut tokens and downstream complexity.
- Why fail over between providers like Anthropic and OpenAI?
- Relying on one provider means a single outage or rate limit can take your feature down. Routing through an interface that can switch providers keeps the feature available, and as a side effect lets you send each task to the cheapest model that meets your quality bar.
- How do you cut LLM cost without hurting quality?
- By measuring quality with evals before and after each change. Evals turn quality into a number, so you can switch to a cheaper model, prune a prompt, or reduce tokens and prove the output did not regress. Without that signal, cost cuts are guesses.
- How much does prompt caching actually save?
- On Anthropic, a cache read bills at 0.1x the base rate, while a cache write costs 1.25x for a five-minute lifetime or 2x for an hour. On one Claude Code request I measured, 93,438 of 93,780 input tokens were cache reads, which prices out near 10,026 base-equivalent tokens rather than 93,780. Cheap is not free: every fresh session pays the write again to lay the prefix down.
- Does switching to a bigger model fix a failing AI feature?
- Often it does not, and it is the expensive reflex. In a bake-off on my own eval fixtures, Sonnet 4.6 at high effort returned 10 of 10 valid outputs at $0.029 each, while Opus 4.8 returned 8 of 10 at $0.073 with no measurable quality gain. The one fixture every model failed was a prompt gap, and a single line of prompt closed what Opus at maximum effort could not.
- What should you measure before cutting LLM cost?
- Find out which part of the request is actually expensive, and at which rate it bills, before you cut anything. Token counts and dollars are different units: a large payload sitting ahead of a cache breakpoint can dominate your context window while barely touching your invoice. Cutting the wrong one costs effort and buys nothing.
Paying too much for an AI feature?
I help teams cut LLM costs without giving up quality. If that is on your plate, get in touch.