Beyond Ng's LangChain Course: Production Lessons

Andrew Ng's DeepLearning.AI LangChain courses are a good on-ramp. They teach you chains, agents, memory, retrievers, and give you enough Jupyter muscle memory to build a demo in an afternoon. I've recommended them to junior engineers on my team. But every time I've shipped a LangChain-based system to a paying customer, the notebook code has been the easy 10%. The other 90% is what nobody teaches you in a course: what happens on Tuesday at 3am when the OpenAI API returns a 529, your Pinecone index is stale, a user pasted a 400KB PDF into the chat, and your monthly bill just jumped from $200 to $4,100.

This is the companion post I wish existed when I started. It assumes you've done the courses and now you're staring at a production ticket.

The notebook-to-production gap nobody warns you about

The single biggest failure mode is treating your LLM app like a normal web service. It isn't. It's a system that calls a non-deterministic, rate-limited, occasionally-lying third-party API on the hot path of every user request, and charges you per token both ways.

That reframing changes everything. In a normal API, latency is measured in ms and errors are exceptions. In an LLM app, p95 latency is 8-12 seconds, "errors" include perfectly successful 200 responses that contain hallucinated JSON, and cost per request can vary 20x depending on what the user typed.

Concretely, here's what shifts when you go from Ng's notebook to production:

The course teaches you the top row. Everything below it is on you.

Retries, timeouts, and the fallback ladder

The first thing I add to any LangChain project going to production is a proper retry and fallback layer. LangChain's built-in with_retry is fine for a demo, but it doesn't distinguish between the errors that matter.

Here's the taxonomy I actually handle:

1. Transient network errors (connection reset, DNS blip): retry immediately, up to 2 times.
2. 429 rate limits: exponential backoff with jitter, respect retry-after header, up to 5 attempts.
3. 529 / overloaded (Anthropic returns these under load): back off longer, and after 2 tries, fall back to a different provider.
4. 5xx from the provider: same as 529.
5. Context length exceeded: no retry, truncate or summarize and try once more, then fail loudly.
6. Content policy refusals: no retry, log with the prompt hash, surface to the user cleanly.
7. Successful 200, malformed output: retry once with a "your last output was invalid JSON, please try again" nudge, then fail.

Category 7 is the one nobody in the course covers. It's also the most common in production. I've had Claude return valid JSON 99.4% of the time in benchmarks and then hit a stretch where 8 of 20 calls came back with a trailing comma. Instructor / structured outputs help, but you still need the guard.

The fallback ladder I use most often on client work:

primary:   claude-sonnet-4.5   (best quality, cheapest for what it does)
fallback1: gpt-4.1             (different provider, different outage window)
fallback2: gpt-4.1-mini        (last resort, degraded quality, still answers)

Two different providers means an OpenAI outage doesn't take you down. I learned this the hard way in November 2023 and again in 2024. Multi-provider is not optional if your SLA promises anything.

Timeouts: hard-cap every LLM call at 60 seconds. Streaming calls at 120s total, with a 20s idle timeout on the stream itself. LangChain's default is infinite, which is how you end up with Lambda functions dying at the 15-minute mark holding an idle socket.

Cost control is a first-class feature, not a monitoring problem

I've seen teams treat their OpenAI bill as something the finance team looks at monthly. That's how a single buggy agent loop turns into a $12,000 surprise. Cost has to be enforced in the request path.

The pattern I ship every time:

• Per-tenant token budget, tracked in Redis, decremented on each call, resets daily.
• Per-request hard cap on input tokens (count with tiktoken before you call, refuse if over).
• Per-request hard cap on max_tokens for output. Don't let the model write a novel because a user asked "summarize this."
• Model routing by task, not by default. Classification and extraction go to a small model. Reasoning and synthesis go to a large one. Ng's course happily uses gpt-4 for everything; in production that's a 30x cost multiplier for no quality gain on 70% of your traffic.

A concrete number from a client SaaS I built last year: routing 68% of calls to a smaller model based on a lightweight intent classifier cut inference spend from about $2,400/month to $780/month with no measurable drop in user-facing quality (measured by thumbs-up rate and human eval on a 200-sample weekly audit).

Also: cache aggressively. Semantic cache for RAG answers with a similarity threshold around 0.93, exact-match cache for structured extraction. On one content pipeline the cache hit rate settled at 41%, which is 41% of the bill gone.

Observability: traces, not print statements

The LangChain course teaches verbose=True. That's fine for one call. In production you need real tracing.

I use LangSmith or Langfuse depending on the client's stack and appetite for self-hosting. Both give you the thing that actually matters: a tree view of every step in a chain or agent run, with inputs, outputs, tokens, latency, and cost per node. When a customer reports "the agent gave me a weird answer at 2:14pm," you need to be able to load that specific trace and see what the retriever returned, what got stuffed into the prompt, and what the model produced.

The minimum I instrument on every project:

• Trace ID propagated from the HTTP request through every LLM call, tool call, and DB query.
• Structured logs with tenant_id, user_id, trace_id, model, prompt_version, input_tokens, output_tokens, cost_usd, latency_ms, outcome (success / retry / fallback / fail).
• Semantic metrics, not just system metrics. "5xx rate" is useless. "Rate of JSON parse failures per prompt version" is the metric that tells you when your prompt regressed.
• Alerting on drift: if the average tokens-per-response for a given endpoint jumps 40% week over week, that's a signal something changed (a prompt, a model, an input distribution). Investigate before the invoice arrives.

When LangChain earns its keep, and when to drop it

Honest opinion after shipping half a dozen production systems: LangChain is great for prototyping and often the wrong abstraction for production.

Where it earns its keep:

• Rapid prototyping when you're still figuring out the shape of the problem.
• Access to a huge library of integrations you'd otherwise write yourself (vector stores, loaders, retrievers).
• LangGraph specifically, for stateful multi-step agents where the graph model actually maps to your problem. LangGraph is the piece I still reach for.

Where I've ripped it out:

• High-throughput single-shot inference endpoints. The overhead of chain construction, callbacks, and object graph traversal adds measurable latency (50-200ms per call in my benchmarks) for something that's just "prompt in, response out." A 30-line async wrapper around the raw SDK is faster, easier to debug, and easier to hand off to another engineer.
• Anywhere the abstraction fights you. If you're subclassing three LangChain classes to get the behavior you want, you've lost. Write it directly.
• Cost-sensitive extraction pipelines. Instructor with Pydantic on top of the raw SDK is cleaner than LangChain's structured output helpers, and you can see exactly what's being sent.

The rule I use: prototype in LangChain, benchmark against a raw-SDK version once the design stabilizes, and keep whichever is simpler for the team maintaining it in six months. For most production endpoints, that's the raw SDK plus a small internal library. For orchestrated agent workflows, LangGraph stays.

Prompts are code, so treat them like code

Ng's course puts prompts in strings inside notebook cells. In production that's untenable. Prompts are the most important, most changeable, most easily-broken part of your system.

What I do on every project past week one:

1. Prompts in files, versioned with the code. I use plain .md or .jinja files loaded at startup, with a version tag baked in.
2. A prompt registry with a stable ID per prompt (rag_answer_v7, intent_classify_v3). The version is logged on every call.
3. An eval set of 30-200 real examples with expected outputs (or a rubric). Run it in CI before merging any prompt change. This is the single highest-ROI practice I've adopted. It catches regressions the moment they happen instead of when a customer complains.
4. Gradual rollout for high-traffic prompts: route 5% of traffic to the new version, compare metrics for 24 hours, then flip.

The eval set doesn't need to be fancy. For a RAG system, I score answers on faithfulness (did it stick to the retrieved context?), relevance (did it answer the question?), and format (did it follow the required shape?). Half of those can be scored by another LLM call, half need a human once a week. That's enough signal to catch 90% of regressions.

What I'd do if I were starting a production LLM project tomorrow

If I were kicking off a new client project on Monday, here's the stack and the order:

1. Week 1: Prototype in LangChain / LangGraph in a notebook. Get end-to-end working with dummy data. Don't optimize anything.
2. Week 2: Build the eval set. 50 real examples minimum. This is non-negotiable and it's the step most teams skip.
3. Week 3: Wire up tracing (LangSmith or Langfuse) and structured logging with trace IDs. Add per-request cost calculation.
4. Week 4: Add the retry / fallback layer. Two providers minimum. Timeouts on everything.
5. Week 5: Move prompts to versioned files with a registry. Wire the eval set into CI.
6. Week 6: Benchmark the hot path against a raw-SDK rewrite. Keep whichever is simpler.
7. Week 7: Add cost budgets, per-tenant quotas, and semantic caching.
8. Week 8: Load test with realistic input distributions. Watch the p95 latency, the token distribution, and the cost per request. Fix whatever surprises you.

Then, and only then, ship to real users. And keep the eval running on a schedule against production traffic samples forever after.

The DeepLearning.AI courses give you the vocabulary and the confidence to start. Getting to production is a separate craft, and it's mostly the boring engineering work that any senior backend engineer would recognize: retries, timeouts, budgets, observability, versioning, gradual rollout. LLMs don't change those fundamentals. They just make ignoring them more expensive.

If you're in the middle of moving an LLM prototype to production and something in here maps to a problem you're hitting, I'm happy to talk. You can reach me at lazar-milicevic.com/#contact, or dig through more field notes on the blog.