How I Run AI PoCs That Actually Ship to Production

Most AI proof-of-concepts I get called in to review have the same problem: they work on the happy path, impress in a demo, and then die in the gap between "cool" and "deployable." The team spent three weeks on prompt tuning and zero on what happens when the LLM returns malformed JSON at 2 a.m. on a Sunday. After a decade of shipping autonomous systems, my rule is simple: a PoC is not a demo, it is the first honest version of the production system. Here is the exact intake-to-handoff process I use to keep the 2-week build alive past the first stakeholder meeting.

What "PoC that ships" actually means

A shippable PoC is a working slice of the real system, running on real infrastructure, against a real subset of data, with real success metrics wired in from day one. It is not a Jupyter notebook, it is not a Streamlit app on someone's laptop, and it is not a Figma flow with prompts pasted in. If the code cannot be promoted to production by adding scale, auth, and observability (not by rewriting), it is not a PoC, it is a sketch.

I hold every PoC to three tests before I call it done:

1. It runs on the same class of infrastructure the production system will run on (usually serverless AWS or a managed container).
2. It has evaluation numbers, not vibes. Precision, recall, latency p50/p95, cost per run.
3. A different engineer can pick it up on Monday and extend it without a knowledge transfer call.

Most PoCs that end up as "discarded demos" fail test 1 or test 3. They ran once, on someone's machine, and no one wrote down why the prompt looks the way it does.

Step 1: The 90-minute intake that decides everything

Before I write a line of code, I run a structured 90-minute intake with the technical sponsor and one domain expert. This is where 80% of PoC failures get prevented. The output is a one-page document I call the PoC contract.

The contract has six fields, and I will not start work without all six:

The single most important field is the last one. If no one owns the production version before the PoC starts, the PoC will be discarded regardless of quality. I have watched excellent builds get shelved because "we will figure out ownership later" is the same as "no one wants this."

I also ask one uncomfortable question: what does the sponsor's org actually reward? If the answer is "shipping features," a PoC has a chance. If the answer is "avoiding blame," even a working PoC will sit in a repo forever. That is a signal to reshape the engagement or walk.

Step 2: Ruthless scope for a 2-week clock

Two weeks is the sweet spot for a first PoC. Less, and you cannot get past prompt-tuning theater. More, and you start building things you do not know you need. Here is how I carve up the days:

• Days 1-2: Data pipeline. Get real data into the system, cleaned enough to work with. This is almost always the biggest surprise. If the PoC deals with documents, you spend Day 1 discovering that "our knowledge base" is actually four SharePoint sites, two Notion workspaces, and a folder someone maintains in Google Drive.
• Days 3-5: Core loop. The minimum viable version of the agent, RAG chain, or workflow. Rough prompts, rough retrieval, no optimization.
• Days 6-8: Evaluation harness. This is what separates shippable PoCs from demos. I build a small eval set of 30-100 real inputs with expected outputs, and a script that runs the system against them and reports scores. I would rather have a mediocre model and a great eval harness than the reverse.
• Days 9-11: Iterate on the numbers. Now the prompt tuning, retrieval tuning, and model swaps have signal. Every change is measured against the eval set. No "it feels better" allowed.
• Days 12-14: Production packaging. Observability, error handling, cost logging, deployment to the target infra, handoff docs.

Notice that only 3 days go to the fun part. That is intentional. In every PoC I have shipped, the difference between a demo and a production system was made in Days 6-14, not Days 3-5.

Step 3: The tech stack choices I actually make

I get asked about stack all the time, so here is the honest version. I default to boring, and I only add complexity when the eval numbers demand it.

LLM: Claude Sonnet for almost everything. Claude Opus when I need reasoning depth on complex agent chains and the cost math works. GPT-class models when the client's org has a hard preference or existing contracts. I benchmark all three on the eval set in Days 6-8 anyway.

Retrieval: Postgres with pgvector plus full-text search combined via Reciprocal Rank Fusion. I wrote a whole post on why this beats "just use a vector DB" for 90% of business RAG cases. Managed Postgres (Supabase, RDS) covers PoC and production without a migration.

Orchestration: I start with plain Python or TypeScript and explicit function calls. I add LangGraph or a custom state machine only when the agent has real branching that benefits from a graph abstraction. Most "agent" PoCs are actually 3-4 tool calls in sequence, and a framework adds more friction than value.

Infra: AWS Lambda + EventBridge + API Gateway for anything event-driven. It scales to zero, costs pennies during PoC, and does not need a rewrite for production. If the workload needs persistent state or GPU, I use ECS Fargate or a managed inference endpoint.

Observability: Structured logs to CloudWatch from day one, with a trace ID on every LLM call. Cost per request logged as a metric. I add Langfuse or a similar LLM-observability tool if the client wants a UI over traces, but I do not depend on it.

The pattern I avoid: piling on tools because a blog post said to. A PoC that uses 8 SaaS services is a PoC that will not get security approval to reach production.

Step 4: The evaluation harness (this is the whole game)

If you take one thing from this post, take this. Build the eval before you build the system. Here is roughly what mine looks like in Python:

import json
from statistics import mean

def run_eval(system_fn, eval_set_path: str) -> dict:
    with open(eval_set_path) as f:
        cases = json.load(f)

    results = []
    for case in cases:
        output = system_fn(case["input"])
        results.append({
            "id": case["id"],
            "expected": case["expected"],
            "actual": output["answer"],
            "score": score_case(case["expected"], output["answer"]),
            "latency_ms": output["latency_ms"],
            "cost_usd": output["cost_usd"],
            "sources_used": output.get("sources", []),
        })

    return {
        "accuracy": mean(r["score"] for r in results),
        "p95_latency_ms": sorted(r["latency_ms"] for r in results)[int(len(results)*0.95)],
        "avg_cost_usd": mean(r["cost_usd"] for r in results),
        "failures": [r for r in results if r["score"] < 0.5],
    }

The score_case function depends on the task. For extraction, exact match on fields. For classification, label match. For open-ended QA, I use an LLM-as-judge with a rubric plus spot checks by the domain expert. LLM-as-judge is not perfect but it is directionally correct at scale, and I calibrate it against 20 human-scored cases first.

The eval set has to include the failure modes the sponsor actually fears. If they said "we cannot invent policies," I include 10 cases about policies that do not exist and check the system says "I do not know" rather than confabulating. This is how you turn "trust me it works" into "here are the numbers on the exact cases you are worried about."

Step 5: The production handoff checklist

The 2-week PoC ends with a handoff, not a demo. Here is the checklist I actually walk through with the receiving team:

• [ ] Repo has a README that runs from a fresh clone in under 15 minutes
• [ ] Infrastructure is defined as code (Terraform or CDK), deploys to a dev account with one command
• [ ] Secrets are in AWS Secrets Manager or the client's equivalent, never in code
• [ ] Every LLM call has: trace ID, input, output, model, tokens in/out, cost, latency, timestamp
• [ ] Eval script runs in CI, fails the build if accuracy drops below a threshold
• [ ] There is a runbook for the top 3 failure modes: model outage, retrieval returning nothing, malformed output
• [ ] Costs are tagged so the team can see spend per feature in the AWS bill
• [ ] A written decision log explains why we chose the model, the retrieval approach, and the prompt structure
• [ ] The eval set is versioned in the repo, not in a Google Sheet

The decision log is underrated. Six months later, when the receiving team wonders "why are we using Claude Sonnet instead of Haiku here?", the log tells them: "Haiku scored 68% on the eval set vs Sonnet at 87%, and the cost delta at expected volume was $180/month, which the sponsor approved on 2026-03-12." That single artifact is what lets a team own the system after I am gone.

Common ways PoCs die (and how I avoid them)

• Data access delayed until week 2. I now refuse to start until I have working credentials and a sample export. "We will get you access next week" means the PoC will run over budget.
• Sponsor changes the success metric mid-build. The PoC contract prevents this. If the metric changes, we reset the clock and rescope.
• No production owner. Covered above, but it bears repeating. If ownership is fuzzy, I insist on naming a person or I do not take the work.
• Building for the average case only. The demo works on 8 out of 10 examples the sponsor picked. Production fails on the 200 weird cases the sponsor never mentioned. The eval set has to be built from real, sampled inputs, not curated ones.
• Choosing an exotic model or framework because it is new. I lost count of the PoCs I have inherited that used a bleeding-edge tool that got deprecated three months later. Boring wins.

What I'd do if you are running your first AI PoC

Start with the PoC contract, not the code. Spend Day 1 arguing about the success metric until everyone in the room can state it in one sentence. Build the eval harness in Week 1, not Week 2. Deploy to the target infrastructure from Day 1, even if the app does nothing yet, so the last day is not a scramble to figure out IAM roles.

And keep the scope brutally small. A PoC that answers one question well is worth ten times a PoC that gestures at ten questions. You can always add the second use case in the next sprint. You cannot un-ship a demo that lost the room by trying to do too much.

If you are scoping an AI proof-of-concept and want a second opinion on the contract, the eval, or the stack, I am happy to look at it. You can reach me at lazar-milicevic.com/#contact, or read more on the blog where I write up the systems I build in production.