Inside an AI Implementation That Actually Lasts

Most AI projects I get pulled into as a second opinion look great in a Loom video and fall apart the moment a real user hits them for a week straight. The demo answered five curated questions. Production has to answer five thousand, half of them malformed, at 2am, with an auditable trail and a bill nobody has to explain to the CFO. This post is the pattern I actually ship: a RAG pipeline plus an agent layer on AWS, with the guardrails, observability, and handover documents that let a client own it after I walk away.

If you are a CTO or head of engineering evaluating vendors, this is the shape of "done" you should be asking for.

The reference architecture I keep coming back to

Ninety percent of the LLM systems I build for clients fit one shape: an ingestion pipeline that turns their messy content into a searchable index, a retrieval layer that does hybrid search with reranking, and a thin agent layer that decides what to do with what it found. Everything runs serverless on AWS so it scales to zero when nobody is asking questions.

Here is the concrete stack I default to, and why:

That is the skeleton. What separates an implementation the client can actually own from a demo they cannot maintain is what wraps around it: the guardrails, the eval suite, the cost controls, and the docs. That is the rest of this post.

Guardrails that survive real users

The two failure modes I see in production are prompt injection through retrieved content and agents that happily call tools they should not. Both are cheap to prevent and expensive to fix after an incident.

Input side. I always separate the user turn from retrieved context using explicit structural tags in the prompt, and I strip or escape anything that looks like an instruction inside retrieved docs. It is not bulletproof but it filters out the low-effort attacks that make up most of what you actually see in logs.

Tool side. Every tool the agent can call has a hard scope check before it runs. A "send email" tool does not just trust the model's arguments. It validates that the recipient belongs to the tenant that owns the session, that the content passes a lightweight classifier, and that the tool has not been called more than N times in this conversation. This is the pattern I wrote about in "One API Key, Five Agents" but the short version is: assume the model will sometimes be wrong and put the blast radius somewhere small.

def send_email_tool(args, ctx):
    # Scope check before the LLM's arguments are trusted
    if not tenant_owns_recipient(ctx.tenant_id, args["to"]):
        return {"error": "recipient_out_of_scope"}
    if ctx.tool_calls_this_turn["send_email"] >= 2:
        return {"error": "rate_limited"}
    if classify_intent(args["body"]) == "suspicious":
        return {"error": "content_blocked", "review_id": queue_for_human(args)}
    return actually_send(args)

Output side. For anything that ends up in front of a user or another system, a schema check. Pydantic or Zod, whatever the language is. If the model returns malformed JSON three times in a row on the same request, the request fails loudly instead of silently returning garbage. Loud failures are how you find real bugs.

Observability that answers the questions you will actually get asked

Six weeks after go-live, someone will ask one of these questions:

1. Why did this specific user get a bad answer on Tuesday at 3:14pm?
2. How much did we spend on LLM calls last month, broken down by feature?
3. Which retrieved documents most often lead to wrong answers?
4. Is quality getting better or worse since the last prompt change?

If your observability cannot answer all four in under five minutes, it is not observability. It is a log stream.

What I actually wire up:

• Per-request traces in Langfuse (or Arize, or Braintrust) covering the full chain: user input, retrieved chunks with scores, rerank order, prompt sent to the model, model output, tool calls, final response. One trace ID that follows the request everywhere, including CloudWatch and the client's app logs.
• Cost attribution tags on every LLM call: tenant, feature, environment, prompt version. Cost tracking that is not tagged is cost tracking you cannot act on.
• Sampled human review. Five percent of production traces get queued into a lightweight review UI where a domain expert clicks thumbs up or down and leaves a note. That labeled data becomes the next round of eval cases. This is how the system gets better over time instead of drifting.

The self-hosted Langfuse deployment sits behind the client's VPC. It is one Postgres and one container. Nobody has ever complained about the ops cost, and everyone eventually thanks me for it when something weird happens.

Cost controls that do not require weekly attention

Left alone, an agent system will drift up in cost. New features add tool calls, prompts grow, someone bumps max_tokens because they were debugging. Six months in you look at the bill and it has quietly doubled.

The controls I put in from day one:

1. A per-tenant monthly budget enforced at the API Gateway layer. When a tenant crosses 80% of budget, the system starts routing to a cheaper model tier and emits a warning. At 100% it returns a clean error with a support link. Never a surprise invoice.
2. A model router that picks the cheapest capable model for the task. Classification and routing questions go to Haiku or a small local model on Ollama for anything running inside the client's infra. Only the actually-hard reasoning steps go to Sonnet or Opus. I wrote about the 40% savings from this pattern separately; it is not clever, it is just discipline.
3. Aggressive prompt caching wherever the model supports it. System prompts, tool definitions, and stable context blocks get cached. On a chat feature with long system prompts, this alone cut input token cost by around 60% in one recent build.
4. A daily cost report posted to Slack, broken down by feature and tenant. If you cannot see the number every morning, you will not notice it climbing.

The rule I give clients: any cost optimization that requires a human to remember to do it will fail. Bake it into the system or do not bother.

The eval suite is the contract

This is the part vendors skip and it is the single biggest tell that an implementation will not last. Without evals, "the AI is broken" is an unfalsifiable claim and every prompt change is a coin flip.

What I hand over:

• A golden set of 100-300 real examples drawn from the client's own domain, with expected answers or expected behaviors. Built collaboratively with a domain expert during weeks two and three of the engagement, not scraped from a benchmark.
• A tiered evaluation runner that scores each example on multiple axes: factual correctness (LLM-as-judge with a rubric, plus exact-match where possible), citation accuracy (did it cite a real chunk that actually supports the claim), safety (did it refuse when it should have), and latency and cost.
• A regression gate in CI. Any prompt or retrieval change runs the full eval suite. If any tier drops more than a defined threshold, the PR does not merge without a written justification.
• A monthly recalibration ritual. New examples get added from the human review queue. Old examples that no longer reflect real usage get retired.

The eval suite is what lets a client's own team confidently change the prompt after I am gone. Without it, they will freeze the system in amber because nobody wants to be the person who broke it.

The handover: what "done" actually looks like

An implementation is not done when the code works. It is done when someone else can own it. The handover package I deliver every time:

A runbook, not a README. Every alarm has a corresponding page that says: this fired, here is what it means, here are the three things to check first, here is who to escalate to. Written in a voice that assumes the reader is on-call at 3am, not in a training session.

An architecture doc with a real diagram. Not a marketing diagram. The actual boxes, the actual arrows, the actual IAM boundaries, the actual failure modes. Updated as part of the definition of done for any change.

A prompt registry. Every prompt version, what changed, what the eval numbers were before and after, who approved it. Git works fine for this. What matters is that "why is the prompt like this" always has an answer.

A model and vendor swap plan. Which models can be swapped for which, what breaks if the vendor changes their API, what the fallback is. Vendor lock-in is fine as long as it is a conscious choice with a documented exit.

A two-hour recorded walkthrough with the client's engineering lead. Screen share, real system, real questions. Kept in their Notion or Confluence. This is worth more than 100 pages of docs.

A 30-60-90 day support agreement where I am available for questions and code review, but the client's team is doing the work. This is the part that turns a delivery into an actual transfer of ownership.

What I would do if I were you evaluating a vendor

Ask three questions and watch the answers carefully.

1. "Can I see your eval suite for a past project?" If the answer is vague or the suite is 10 hand-picked prompts, you are buying a demo. A real practitioner will show you a numbered dashboard and talk about regression thresholds.
2. "Walk me through what happens when a user reports a bad answer." Anyone who cannot describe the path from user complaint to reproduced trace to identified failure mode to eval case to fixed prompt in under two minutes has never actually operated one of these systems.
3. "What does your handover include?" If the answer is "documentation and a walkthrough," push harder. Ask about runbooks, alarms, the prompt registry, the model swap plan, and the support tail. The gap between vendors on this question is enormous.

The systems that last are boring in the best way. They have small, well-scoped components. They fail loudly. They tell you what they cost. They can be changed with confidence. None of this requires exotic technology. It requires the discipline to build all the parts nobody sees in a demo.

If you are working through an implementation like this and want a second set of eyes, or you are earlier and trying to figure out whether to build in-house or bring in help, I am reachable at lazar-milicevic.com/#contact. There is more on how I approach production LLM systems on the blog.