What Billions of Predictions Taught Expedia About AI at Scale

I keep coming back to a line from Expedia's engineering leadership: the hard part of AI at scale is not shipping the model, it's keeping it alive. That matches everything I've learned running autonomous content systems and serverless AI in production. The first working version is the easy part. Month six, when your data has drifted, your prompts have quietly regressed, and nobody remembers which version of the retriever is actually live, is where reality hits.

This post is about what separates AI that works today from AI that lasts. I'll pull from Expedia's public lessons on running billions of ML predictions and pair them with what I've seen building AI agents, RAG pipelines, and self-improving content loops for real workloads.

The scale problem is not the model, it's the surface area

Expedia has talked openly about serving billions of predictions across ranking, pricing, personalization, and fraud. The interesting part is not the throughput. It's the fact that dozens of models feed each other, share features, and depend on data pipelines that were built by different teams at different times. That is the surface area that kills you.

The same shape shows up in my own work at a much smaller scale. A "simple" content agent that researches, drafts, optimizes for search, and publishes is really:

• 3 to 5 LLM calls with different prompts and models
• 2 retrieval layers (pgvector plus a keyword index, fused with RRF)
• 4 external APIs (search, CMS, analytics, image generation)
• 1 scheduler and a queue
• 1 eval harness that grades outputs before publish

Each of those is a place where behavior can silently change. A model version rolls forward. An API changes a default. A pgvector index is rebuilt with a different distance metric. A prompt gets "improved" by someone who forgot the eval set. Any single change can be shipped in an afternoon. The compound effect over a quarter is a system that no longer does what its owners think it does.

Velocity without discipline is not speed. It's debt with interest.

Drift is the boss fight, not the tutorial

The most expensive bugs I've had in production AI systems were not crashes. They were quiet regressions. The system kept running. Metrics looked fine at a glance. Outputs got worse. That is drift, and it comes from three directions at once.

Data drift. The distribution of inputs changes. In travel that's obvious: search intent shifts by season, event, and geopolitics. In my content systems it's less obvious but just as real: what people search for on "AI agents" in July 2026 is not what they searched for six months ago, and a retriever tuned on last quarter's corpus starts pulling stale context.

Model drift. You didn't change the model. The provider did. Anyone who has run Claude or GPT calls in production for more than a year has felt this. Same prompt, same temperature, subtly different behavior. Sometimes better, sometimes worse, almost never announced with enough specificity to know.

Prompt drift. This one is self-inflicted. A prompt gets tweaked to fix one bad case, and three months later nobody can reconstruct why it looks the way it does or which of the seventeen instructions are actually load-bearing.

The Expedia lesson here, and mine, is the same: you cannot fix drift by being careful. You fix it by measuring continuously. Offline evals at commit time. Online evals on live traffic. Golden sets that never change. Shadow deploys before promotion. If that infrastructure does not exist, you are not running AI in production. You are running AI in hope.

The eval harness is the product

I've come to believe that for any serious AI system, the eval harness is more valuable than the model. Models are replaceable. A good eval set, with real examples, real labels, real edge cases, and a reproducible scoring pipeline, is the thing that lets you swap models without fear. It's the asset.

Here's the minimum shape I run for a production LLM feature:

The LLM-as-judge layer is where most teams get burned. A judge that isn't itself evaluated is just another model quietly drifting. I calibrate judges against human labels on the golden set every time I change the judge prompt or model. If judge-human agreement drops below the threshold I set (usually around 85% for pass/fail, lower for graded scores), the judge doesn't ship.

If you want the longer version of how I run this, I've written about LLM evals in production and how it compares to Hamel Husain's approach on the blog. The short version: build the harness first, then the feature. Not the other way around.

Feature stores and prompt registries: boring wins

Expedia has been public about investing in a feature platform so that models across teams share consistent, versioned inputs. The lesson is not "you need a feature store." It's that shared, versioned, observable primitives are what let AI systems compound instead of collide.

For LLM-heavy systems, the equivalents I actually use:

• Prompt registry. Every prompt is a file with a version, an owner, a linked eval set, and a changelog. Not a string in application code. When something regresses, I can git blame a prompt.
• Retrieval config as code. Chunk size, overlap, embedding model, distance metric, top-k, rerank config, RRF weights. All versioned. All logged with every request. If you ever want to know why a specific answer looked the way it did on a Tuesday in April, you need this.
• Model routing table. Which task uses which model at which temperature with which max tokens and which fallback. One file. Reviewed like any other config.
• Trace store. Every LLM call: inputs, outputs, tokens, latency, tool calls, retrieval hits, cost. I use structured logs to a Postgres table for small systems and a proper trace backend for larger ones. Without this, you cannot debug anything.

This sounds like overhead. It is overhead. It's also the difference between a system you can evolve and a system you're afraid to touch.

Cost discipline is not a finance concern, it's an architecture concern

Billions of predictions cost real money. So do billions of tokens. I've seen small agent systems run away to five-figure monthly bills because a single retry loop wasn't capped, or a context window was being padded with irrelevant history.

The two levers that actually move cost in LLM systems:

1. Route by difficulty. Cheap model for the 80% of easy cases, expensive model for the hard ones. Classify first, then answer. On one content pipeline I cut token spend by roughly 60% just by routing draft generation to a smaller model and reserving the frontier model for the final polish and eval.
2. Cut context aggressively. Most agents carry way more history than they need. I've written about reducing agent token use with tool routing and how careful memory design took one system from 3.26M to 118K tokens per session with no quality loss. Same idea applies to RAG: retrieve less, rerank better.

The pattern here mirrors what large-scale ML teams have always done: measure per-request cost, budget it, alert on it, and treat a cost regression the same as a latency regression. If your dashboard doesn't show cost per successful outcome, you're flying blind.

Data discipline is the moat

The single most underrated lesson from ML at scale is this: the team that owns clean, well-labeled, well-versioned data wins. Not the team with the best model. Not the team with the best prompt. The team whose training and eval data actually reflects reality and stays that way.

For AI agents and LLM apps, "data" means:

• Real user interactions, captured with consent and structure
• Labeled failures, not just labeled successes
• Edge cases, adversarial inputs, and known-hard examples, preserved forever
• Ground truth for retrieval: for a given query, what documents should have come back
• Outcome data: did the AI's answer actually help the user or the business

I've watched teams pour months into prompt engineering and model selection while never once sitting down to label 200 real examples of their own product's failures. That labeling session is almost always the highest-leverage day of the quarter. It surfaces problems no benchmark will ever catch and gives you an eval set that is genuinely yours.

The self-learning content loop I run for BizFlowAI works precisely because of this. It measures real search performance, feeds those signals back into targeting and generation, and improves from actual outcomes rather than assumed ones. It is not a smarter model. It is a tighter feedback loop.

What I'd do if I were starting a new AI system tomorrow

Concrete, opinionated, in order:

1. Write the eval harness before the feature. 50 real examples, hand-labeled, checked into the repo. Make it runnable with one command. This is week one, not month three.
2. Instrument every LLM call from day one. Inputs, outputs, model version, prompt version, retrieval config, tokens, cost, latency. Structured logs to a table you can query.
3. Version everything that changes behavior. Prompts, retrieval configs, model routing, judge prompts. Nothing behavior-affecting lives as an unversioned string.
4. Ship the smallest useful thing behind a flag. Route 5% of traffic. Compare against the current baseline on your eval set and on live metrics. Only then ramp.
5. Calibrate your judge against humans. Every time you change it. If agreement drops, the judge doesn't ship.
6. Budget cost per outcome, not per call. Alert when cost per successful task changes by more than 20%.
7. Set a weekly drift review. 30 minutes. Look at eval trends, cost trends, judge-human agreement, and a random sample of live traces. Boring. Effective.
8. Kill features that don't earn their eval budget. If nobody has time to maintain the eval set for a feature, that feature is not really in production. It's just running.

None of this is exotic. It's the discipline that turns a demo into a system. Expedia's teams learned it the hard way at billions-of-predictions scale. The rest of us can learn it earlier and cheaper, if we're willing to slow down for a week to build the boring parts.

Closing

The industry loves the story of the model that works. The story that matters is the model that still works next quarter, with new data, a new provider release, and a new team member who wasn't there when it was built. That's the system worth building, and it's the one worth paying for.

If you're building AI features that need to survive contact with real users and real time, and you want a second pair of eyes on the eval, drift, and data discipline underneath them, I'm reachable at lazar-milicevic.com/#contact. More field notes on production AI, agents, and RAG on the blog.