What to Look for in a Custom AI Agent Dev Shop

Last month a founder sent me a proposal from an "AI agent development company" quoting $180k for a 12-week build. The scope document was three pages. Two of those pages were logos and case study screenshots. The actual technical section was 400 words and mentioned "LangChain" once, "agent" 14 times, and "eval" zero. I told him to walk.

I run agent builds at BizFlowAI and I've been on both sides of these calls: the engineer explaining why a demo is not a product, and the buyer trying to figure out if the shop across the table has actually shipped one. Here is what I look for, what I ignore, and the checklist I'd hand to any founder before they wire a deposit.

Start with how they scope, not how they sell

The single strongest signal is how a shop scopes a project before they've been paid. A serious agent team will refuse to quote a fixed price on a vague brief. They will push you toward a paid discovery or a small PoC first, because they know what agent builds actually cost when the edge cases show up in week 6.

Here is the scoping conversation I have with prospects at BizFlowAI, and the one you should expect from anyone competent:

1. What does the agent decide, not just do? (Retrieval is not agency.)
2. What is the cost of a wrong decision, in dollars and in trust?
3. What tools does it call, and who owns the auth for each one?
4. What is the human-in-the-loop policy for the first 90 days?
5. What does "done" look like as a measurable eval, not a vibe?

If the shop skips straight to "we'll use LangGraph and Claude Sonnet, here's the price," they are selling you a template. The template might even work. But you are paying custom rates for stock parts, and when your agent hallucinates a refund into a customer's account, the same template won't tell you why.

Ask for a scoping document from a past engagement (redacted is fine). If they can't produce one, they don't have a repeatable process.

Engagement models: what you're actually buying

There are four models in the market, and they solve different problems. Mixing them up is how founders end up with a $200k prototype that nobody owns.

The mistake I see most often is buying a fixed-scope PoC and expecting a production system at the end. A PoC proves a hypothesis. A production system needs evals, observability, retries, cost caps, prompt versioning, a rollback plan, and a human on-call rota. If the proposal doesn't separate these two phases, you are being sold hope.

At BizFlowAI I now refuse to quote production work until a PoC is on the ground. The gap between "the demo works" and "it runs unattended for 30 days" is usually 3x the effort of the demo itself. Any shop that tells you otherwise has not run one in production.

Deliverables that actually matter

A logo on a slide is not a deliverable. Here is what I want to see leaving the repo at the end of an agent engagement, and what I ship on my own builds:

Code and infrastructure:

• A repo with clear boundaries between orchestration, tool layer, prompts, and evals. Not one 2,000-line agent.py.
• IaC for whatever cloud you're on (I default to AWS Lambda + EventBridge for scheduled agents, containers on ECS or Fly for long-running ones).
• Secrets management that is not .env in the repo. AWS Secrets Manager, Doppler, or 1Password Connect.
• Prompt versioning tied to git commits, so you can bisect regressions.

Observability:

• Structured traces for every agent run. I use Langfuse or Arize; a shop that "logs to CloudWatch" for a multi-step agent has never debugged one at 2am.
• Cost per run, latency per tool call, and a dashboard the client can read without the vendor.
• Alerting on failure rate, cost anomalies, and token spikes.

Evals:

• A gold-standard dataset of at least 50-100 real cases from the client's domain.
• Automated eval runs on every prompt or model change, gated in CI.
• Both LLM-as-judge and deterministic checks (regex, schema, tool-call correctness).

Documentation:

• An architecture diagram that matches the code.
• A runbook for the top 10 failure modes.
• A handover doc that lets an internal engineer take over in a week.

If a shop's proposal doesn't list evals as a first-class deliverable, they are building on sand. I've seen agent projects "ship" and then silently degrade for three months before anyone noticed retrieval quality had collapsed after a data migration. Evals catch that in an hour.

Red flags I've learned to spot in the first call

Some of these I've earned by watching client rescues, some by making the mistakes myself when I was greener. In no particular order:

"We'll fine-tune a model for your use case." Almost never the right first move in 2026. For 95% of business agent workloads, better retrieval, better prompts, and structured tool use beat fine-tuning at a tenth of the cost and time. If fine-tuning comes up in the first meeting, they are optimizing for invoice size.

No mention of context engineering. Prompt engineering is table stakes. Context engineering (what goes into the context window, in what order, from which sources, with which compression) is where production agents win or lose. If the shop can't talk about hybrid search with RRF, chunking strategies, or context pruning, they haven't shipped a serious RAG-backed agent.

A framework as the answer to every question. LangChain, LangGraph, CrewAI, AutoGen, Pydantic AI. All fine tools. None of them are a strategy. I've built production agents in raw Python with a state machine and 200 lines of orchestration because the framework tax wasn't worth it. A shop that reflexively reaches for one framework for every problem is not thinking.

No answer for "what happens when the model changes?" Anthropic and OpenAI ship new versions every few months. Behavior shifts. Costs shift. If the shop has no eval harness to re-test on a new model, every provider update is a fire drill.

Offshore team, onshore price, no engineer on the call. Nothing wrong with distributed teams; I work remote out of Belgrade. But if the person you're talking to can't answer a specific technical question ("how do you handle idempotency on tool retries?") without checking with someone else, you are talking to a sales layer, not the builder.

They won't name their stack. A confident team will tell you exactly what they use and why, and where they'd swap it out. Vague answers ("we pick the right tool for the job") after 30 minutes of questions usually mean they haven't picked.

Case studies with no numbers. "We built an agent for a Fortune 500 healthcare company." Cool. How many runs a day? What's the failure rate? What did it replace? If the case study can't produce one honest metric, it's marketing fiction.

The founder's checklist for vendor calls

Print this. Bring it to the call. Score honestly.

Team and process (10 points):

• [ ] The lead engineer is on the sales call, not just the founder or account manager.
• [ ] They can walk through a past project's architecture from memory.
• [ ] They mention evals unprompted in the first 20 minutes.
• [ ] They ask about your data before proposing a solution.
• [ ] They push back on at least one thing you said. (Yes-men ship bad agents.)

Technical depth (10 points):

• [ ] They can explain their choice of orchestration layer with trade-offs.
• [ ] They have an opinion on hybrid search vs pure vector, and why.
• [ ] They talk about cost per run, not just capability.
• [ ] They have a story for observability that isn't "we'll add logging."
• [ ] They discuss human-in-the-loop as a design decision, not a fallback.

Commercials and ownership (10 points):

• [ ] You own the IP, the repo, the prompts, and the eval dataset. In writing.
• [ ] They provide a handover process, not just a delivery.
• [ ] Pricing separates PoC from production explicitly.
• [ ] There is a defined exit ramp if the PoC fails, without extra fees.
• [ ] Payment milestones are tied to deliverables, not calendar dates.

Under 20 out of 30, keep looking. Above 25, worth a paid discovery. The middle band is where most shops sit, and where you need to weigh other factors like domain fit and timeline.

Two questions that separate the operators from the demo-shippers

I'll leave you with the two I use to cut through a pitch fast. Ask these near the end of the first call, when the polished parts of the deck are already spent.

"Walk me through the last agent you shipped that broke in production. What broke, how did you find out, and how long to fix?" A real operator has this story ready and tells it with specifics: a tool call timing out, a context window overflow, a silent quality regression from a model update. The answer reveals their observability, their on-call posture, and their honesty. If they claim nothing has ever broken, they either haven't shipped or won't tell you when they do.

"If I wanted to fire you in month four and take this in-house, what would I have to do?" The right answer is a straight one: here is the repo, here is the runbook, here are the evals, here is a week of handover, we're done. A shop that gets uncomfortable with this question has built in vendor lock-in on purpose. That's fine for some businesses, disastrous for AI systems that need to evolve with your product.

What I'd do if I were hiring a shop today

If I were a founder buying a custom agent build in 2026, here is my exact playbook:

1. Start with a paid discovery, $5k-$10k, one week. Get an architecture doc and a scoping proposal out of it. This filters serious shops from spray-and-pray sales teams cheaply.
2. Run a fixed-scope PoC with two vendors in parallel if the project is above $100k. It's more expensive up front and saves you six months later. Same brief, same dataset, compare the deliverables side by side.
3. Never buy production and PoC in one contract. Structure them as separate SOWs with a clear go/no-go decision point.
4. Insist on eval ownership. The eval dataset is more valuable than the code. Code you can rewrite. Ground truth you cannot regenerate cheaply.
5. Budget 30-40% of the build cost for the first year of operations. Agents drift. Costs shift. The shop that builds it should not be the same team you audit it with 12 months in, but they should hand off cleanly to whoever is.

The market is loud right now. Every consultancy has rebranded as an agent shop. The good ones talk about failure modes, evals, and handover in the first hour. The rest sell you a demo and disappear when the model version changes.

If you're weighing a build and want a second pair of eyes on a proposal, or a scoping conversation before you commit, reach out at lazar-milicevic.com/#contact. I also write more of these engineer's-side-of-the-table notes on the blog, if you want to keep reading before we talk.