Cutting LLM Token Spend 40% Without Losing Accuracy

The first production LLM system I shipped burned through $8,400 in API costs its first month. The demo had cost me $12. Nobody warned me that the gap between "works in a notebook" and "works for 40,000 daily requests" is measured in five-figure invoices. Since then I've spent a lot of time on the boring, unglamorous work of getting that number down without letting quality slip. A recent paper from Writer describing their orchestration "harness" hit a nerve, because the techniques they systematize are the same ones I've been stitching together by hand for the last two years.

Here's what actually works, what the Writer approach adds, and where I'd start if you're staring at an AWS bill you can't defend.

The ROI paradox nobody warns you about

In the pilot phase, you route every request to the strongest model available. Claude Opus, GPT-5, whatever the current top-of-stack model is. It works. The demo goes well. Someone signs a contract. Then you deploy, traffic scales 100x, and your gross margin turns negative before the end of the quarter.

The Writer paper frames this well: enterprise AI economics break not because models are bad, but because teams treat orchestration as an afterthought. Every request gets the same treatment. Every prompt carries the same bloat. Every retrieval dumps the same context window. You are, in effect, paying premium pricing for premium reasoning on requests that a 7B model could handle in 200ms.

In my own systems, I've measured the split. Roughly 60-70% of production requests do not need frontier-model reasoning. They need retrieval, formatting, classification, or extraction. The remaining 30% is where you actually earn the token spend. Once you internalize that split, the optimization strategy writes itself.

Where the money actually goes

Before you optimize, instrument. I cannot stress this enough. Most teams I talk to have no idea where their tokens are going. They know the monthly bill. They don't know the per-endpoint, per-model, per-prompt-template breakdown.

Here's the breakdown I typically see in a mid-scale production RAG or agent system:

The two fat targets are almost always retrieved context and system prompts. If you fix nothing else, fixing those two gets you to a 30% reduction on most systems. Add smart routing on top and you're at 40% or more, which lines up with what Writer reports and with what I've hit on my own projects.

Step 1: Route requests to the cheapest model that can actually do the job

This is the single highest-leverage change. Not every request is worth a Claude Opus call. My routing layer is a small classifier, sometimes a fine-tuned local Llama 3, sometimes just Haiku or GPT-5-mini with a tight system prompt, that decides which downstream model to call.

The taxonomy I use:

• Tier 0 (local, free): intent classification, PII detection, safety filtering, simple extraction. Ollama with a quantized 3B or 7B model on the same box as the API. Zero API cost, sub-100ms.
• Tier 1 (cheap API): structured extraction, summarization, formatting, single-turn Q&A over already-retrieved chunks. Haiku, GPT-5-mini, Gemini Flash. Roughly 1/20th the cost of the flagship.
• Tier 2 (flagship): multi-step reasoning, code generation, tool use with planning, anything the customer actually judges you on. Claude Opus or GPT-5.

The router itself is dumb on purpose. A 40-line prompt, a few-shot rubric, and a hard fallback to Tier 2 if confidence is low. In one autonomous content pipeline I run, this split alone cut monthly spend by 34% and, counterintuitively, improved latency on the majority of requests because the cheaper models are also faster.

The gotcha: you must eval per tier. A model that scores 92% on your golden set at Tier 2 is not the same model at Tier 1. Build a small eval harness (I use a mix of Ragas for RAG-heavy paths and custom LLM-as-judge for open-ended outputs), and track accuracy per route. When Tier 1 accuracy drops below a threshold on a specific intent, bump that intent up.

Step 2: Compress prompts without breaking them

Prompt compression is where the Writer harness idea gets clever. The naive version, "just shorten the system prompt," works up to a point and then breaks in weird ways. The version that actually holds up in production is more surgical.

Three techniques I use, in order of leverage:

1. Separate static from dynamic. Your system prompt has two parts: the instructions that never change (persona, rules, output format) and the parts that change per request (user context, current date, retrieved facts). Cache the static part. On Anthropic's API, prompt caching cuts the repeated portion to 10% of the input token cost. On a 4,000-token system prompt hit 500,000 times a month, that's real money. OpenAI has automatic caching too, but you have to structure the prompt with the stable prefix first for it to trigger.

2. Distill the instructions. Most system prompts I inherit are 3,000+ tokens of accumulated fear. Every past bug fix added a "PLEASE DO NOT" clause. Every stakeholder demand added an "IMPORTANT:" section. Take your current prompt, run it through a compression pass with Claude ("rewrite this system prompt to preserve every behavioral rule but in 1/3 the tokens, then list what you changed"), then eval the compressed version against your golden set. If accuracy holds, ship it. I've cut prompts from 3,800 tokens to 1,100 with zero measurable regression on a customer-support agent.

3. Use LLMLingua-style compression on retrieved context. For long retrieved passages, a small model can compress the passage while preserving the tokens that matter for the query. Microsoft's LLMLingua is the reference implementation. In practice, for a RAG pipeline pulling 8 chunks of 500 tokens each, you can compress the total context by 40-60% with minimal answer-quality loss. The trick is to compress conditional on the query, not blindly.

Step 3: Retrieve less, but retrieve better

Stuffing top-k chunks into context is the laziest thing we do in RAG, and it's the biggest token bleed. If your k is 10 and each chunk is 400 tokens, you're paying for 4,000 tokens of context on every single request, most of which the model ignores.

What I do instead:

1. Hybrid search with RRF fusion. Semantic search alone misses acronyms, product SKUs, and exact-match terms. BM25 alone misses paraphrases. I run both in parallel over pgvector plus a Postgres full-text index, then merge with reciprocal rank fusion. Retrieval quality goes up, which means I can drop k.
2. Rerank aggressively. After the initial retrieval of 20-30 candidates, a cross-encoder reranker (Cohere Rerank, or a local BGE reranker) trims to the top 3-5 that are actually relevant. The reranker adds ~80ms but lets me drop from k=10 to k=4 without losing answer quality.
3. Dynamic k based on query type. A simple factual lookup often needs one chunk. A synthesis question might need five. I have the router predict the required breadth as a side output, and the retrieval layer adjusts. This alone dropped my average context size by 35% on one system.

The result: better answers on fewer tokens. Not a trade-off, a Pareto improvement, once you take the time to build it.

Step 4: Cap the output, and structure it

Output tokens are typically 3-5x more expensive than input tokens. And yet almost no team I audit sets max_tokens aggressively or forces structured output.

Concrete moves:

• Force JSON output when downstream code consumes the result. Structured output kills the "here's your answer, and let me also add some context..." tail that models love. On some endpoints I've cut output tokens by 40% just by switching from prose to a JSON schema.
• Set max_tokens to what you actually need. If your UI shows 3 bullets, don't let the model generate 15 and then truncate. Set max_tokens to something like 250 and design the prompt around it.
• Stream and cut. For chat interfaces, stream the output, and let the client cut off when the answer is complete. Combined with a stop sequence, you can save 20-30% of output tokens on chatty models.

Step 5: Build a feedback loop so this doesn't decay

The uncomfortable truth about token optimization is that it decays. New features add new prompts. Retrieved context grows as your corpus grows. Somebody adds a "just in case" clause to the system prompt. Six months later, you're back where you started.

What I put in place on every production LLM system now:

• Per-endpoint token dashboards. Grafana, one panel per endpoint, tracking input tokens, output tokens, cache hit rate, and model tier distribution. Anomalies trigger Slack alerts.
• Weekly eval runs. Golden set of 200-500 examples per endpoint. Runs every Sunday. Regression on accuracy or cost triggers a ticket.
• Prompt versioning in git. Every prompt is a versioned file. Changes go through review. This alone catches half the accidental bloat before it ships.

Writer's harness formalizes this loop, which is what makes the paper interesting to me. Most teams treat orchestration as glue code. Treating it as a first-class system, with its own eval, versioning, and monitoring, is what turns a 15% cost reduction into a 40% one.

What I'd do this quarter if I ran a production LLM system

If I were parachuted into a team burning money on GPT-5 for everything, here's my 30-day sequence:

1. Week 1: Instrument. Log input tokens, output tokens, model, and endpoint for every call. Build one Grafana dashboard. Find the top 5 endpoints by spend.
2. Week 2: Enable prompt caching on the top 5. Distill their system prompts. Ship. This alone is usually a 20-25% cut with no risk.
3. Week 3: Build a router. Start with a hard-coded rule table (endpoint X always goes to Haiku, endpoint Y always to Opus), then add a small LLM classifier for the ambiguous middle. Eval per route.
4. Week 4: Attack retrieval. Add reranking, drop k, add hybrid search where you don't have it. Compress context with LLMLingua on the longest passages.

By day 30 you're usually at 35-45% cost reduction with equal or better accuracy. It's not glamorous. There's no announcement post to write. But the difference between an LLM product that scales and one that gets shut down for burning cash is exactly this kind of work.

Closing thought

The Writer paper is a useful signal that orchestration is finally getting the attention it deserves. Foundation models are commodities now. The engineering around them, the routing, the compression, the retrieval, the eval loop, is where the real leverage lives, and where the margin between a healthy AI product and an expensive science project is decided.

If you're running an LLM system in production and the math isn't working, I'm happy to take a look. Get in touch at lazar-milicevic.com/#contact, or read more of what I've written about production LLM systems and agent architecture on the blog.