How to Hire an AI Consultant Who Actually Ships

Last month a founder sent me a proposal from another consultant. Forty pages, three architecture diagrams, LangChain everywhere, a Gantt chart. Zero mention of evals, latency budgets, cost per request, or what happens when the LLM returns malformed JSON at 2am. I read it twice to make sure I wasn't missing something. I wasn't.

The gap between an AI implementation consultant who ships production systems and someone who fine-tunes prompts in a Jupyter notebook is enormous, and it is mostly invisible on a resume. This is the buyer's guide I wish more CTOs had before they hired me to clean up after someone else.

What "AI implementation consultant" should actually mean

An AI implementation consultant designs, builds, and hands over production LLM systems that keep running when you stop paying attention. That means a working retrieval layer, deterministic evals, cost and latency tracking, error handling, deploy pipelines, and a runbook. Not a demo notebook, not a slide deck, not a Zapier chain wrapped around an OpenAI key.

The easiest way to spot the difference: ask what happens after the demo. A hobbyist talks about the model. A shipper talks about the boring stuff: retries, idempotency, cache invalidation, structured outputs, observability, the eval harness, the on-call plan. If your consultant cannot describe how they know their system is degrading before a customer complains, they have not shipped one.

A working definition I use with prospective clients:

• Owns the full stack: retrieval (pgvector, FTS, RRF), orchestration (LangGraph or hand-rolled), model calls, structured outputs, storage, queueing, deploy.
• Owns the evals: a real test set, offline + online metrics, regression gates before merge.
• Owns the numbers: cost per request, p50/p95 latency, hit rate, hallucination rate, human review rate.
• Owns the handover: docs, runbook, dashboards, and a working local dev setup your team can actually run.

If a candidate cannot speak fluently to those four, you are hiring a prototyper.

The 30-minute scoping call that saves you six months

Most scoping calls are wasted on vague "AI transformation" chatter. Here is the structure I use, and the one you should demand as a buyer.

Minutes 0-5: The one workflow. Force the conversation to a single, concrete workflow. "We want to use AI" is not a project. "We want to auto-triage inbound support tickets into these 6 categories and draft a reply for the top 3" is. If the consultant lets you stay abstract, they are either inexperienced or they are billing hourly.

Minutes 5-15: Data reality check. Where does the data live? How clean is it? Who owns access? What is the volume per day? What is the historical dataset you can eval against? If there is no ground-truth data, the first phase is not building an agent, it is building the eval set. A good consultant will say this out loud in the first call.

Minutes 15-25: Success criteria. Not "make it better." Specific numbers. If you cannot state the acceptance criteria in one sentence with a threshold, you are not ready to scope. Example: "Classification accuracy >= 92% on a 500-example holdout set, p95 latency < 3s, cost < $0.02 per ticket."

Minutes 25-30: Kill criteria. What would make us stop? Every serious PoC I run has an explicit kill line. If we cannot pass 80% accuracy by week 3, we stop and reassess. Consultants who avoid kill criteria are selling you optionality at your expense.

If a consultant walks out of a 30-minute call with a fixed-price quote, be suspicious. If they walk out with a written one-page scope, three open questions, and a proposed two-week PoC with a kill line, hire them.

Proof-of-concept pricing: what's fair in 2026

PoC pricing is where most engagements go sideways. The market has settled into three reasonable shapes. Anything outside these should raise a flag.

A few honest observations from running these:

• Beware the $2k PoC. Someone is either using you to learn on the job or they are shipping a wrapper around a public template. Real PoC work has real hours in it: eval set curation alone is often 20-30 hours.
• Beware the $200k "strategy" phase. If a firm wants six figures before writing code, they are selling you a deck. The big-consulting slide-and-invoice model is the single biggest reason enterprise AI stalls.
• Fixed price only works when scope is genuinely fixed. If the client cannot describe the input and output in one page, a fixed price is a lie one of you will pay for later.

I personally quote fixed for 2-week spikes and time-and-materials with a weekly cap for anything longer. Weekly caps protect the client from runaway hours and force me to communicate when scope drifts.

Remote vs local: the trade-off nobody names honestly

I have worked remotely with teams in London, Toronto, Dubai, and Berlin for years. I also live in Belgrade and occasionally take on-site engagements when it genuinely matters. The honest trade-off is not about time zones or Slack. It is about decision velocity.

Remote works beautifully when:

• The client has a designated technical owner who can answer questions within a business day.
• Data access is already sorted (or the consultant can build against a representative sample).
• Success criteria are written down.
• There is a working async communication culture (Linear, Notion, Loom, PR reviews).

Local (or hybrid) is worth the extra cost when:

• The problem is deeply entangled with tribal knowledge that only surfaces in hallway conversations.
• Data cannot leave the building for regulatory reasons and access needs to be granted in person.
• The org is early in its AI journey and needs a warm body in strategy meetings.

A useful heuristic: if your team is already remote-native and ships software async, hire the best AI engineer regardless of location. If your team is co-located and struggles with async, either fix that first or budget for a consultant who will fly in for kickoff and mid-project checkpoints.

Time zones matter less than people think. A 4-6 hour overlap window is plenty for a good async workflow. What kills projects is not distance, it is silence.

The technical questions that separate shippers from hobbyists

Here is the interview I actually run when clients ask me to vet another consultant. None of these are trick questions. All of them have wrong answers.

1. "How do you evaluate a RAG system?"

Wrong answer: "We check the outputs and iterate." Right answer names retrieval metrics (recall@k, MRR) separately from generation metrics (faithfulness, answer relevance), mentions a golden dataset, and describes how they gate deploys. Bonus points for mentioning ragas, promptfoo, or a hand-rolled harness with LLM-as-judge and human spot-checks.

2. "Walk me through hybrid search with pgvector."

Wrong answer: "We use pgvector for embeddings." Right answer: dense embeddings in pgvector, sparse via Postgres FTS with tsvector, combined with Reciprocal Rank Fusion, tuned k for each, and a rerank step (Cohere or a cross-encoder) for the top N. If they cannot explain RRF in one sentence, they have not built hybrid search.

3. "How do you handle malformed model outputs in production?"

Wrong answer: "GPT-4 is pretty reliable." Right answer: structured outputs (JSON schema, tool calling), a Pydantic or Zod validator on the way out, a bounded retry with a stricter prompt, and a fallback path that returns a graceful error the calling system can handle. They should mention idempotency keys.

4. "What is your cost per request and how do you track it?"

Wrong answer: silence, or "it depends on the model." Right answer: token counts logged per request, tied to user and workflow, aggregated in a dashboard, with alerts on anomalies. Ideally they can quote a number from a real system they built. When I ran my content pipeline through Claude and OpenAI in parallel, tracking cost-per-published-article to two decimals was what let me actually optimize the prompt chain.

5. "How do you deploy and roll back?"

Wrong answer: "We push to production." Right answer: prompts and model versions are pinned in code, changes go through PR + eval gate, deploy is via CI to a serverless environment (Lambda, Cloud Run, Vercel), and rollback is a one-command redeploy of the previous version. Prompts should be versioned, not edited live.

6. "Show me an agent you built that is running in production today."

This is the one that filters hardest. Ask for a repo, a live URL, or a video walkthrough of a dashboard showing real usage. Certifications and course completions are fine. Shipped systems with logs are the actual signal.

Red flags I have learned to trust

Fast pattern-matching from ten years of watching these engagements go well or badly:

• LangChain in every sentence. LangChain is a fine library. If a consultant cannot describe the underlying HTTP calls and why they chose the abstraction, they are cargo-culting.
• "We use RAG" with no talk of chunking strategy, reranking, or eval. RAG is 20% embedding and 80% everything else.
• No mention of observability. If they cannot show you what a Langfuse, Helicone, or custom trace looks like, they have not run one in production.
• "AI agents will handle it." Multi-agent systems are the current fashion. In production, most problems are solved with one well-prompted LLM call, structured output, retries, and a queue. If they lead with a 12-agent architecture for a summarization task, they are showing off.
• No opinion on when NOT to use AI. Serious practitioners have a short list of problems they refuse to solve with an LLM. Ask for it.
• They quote before they understand your data. Nobody can price a RAG system without seeing (or at least sampling) the corpus.

What I'd do if I were hiring right now

If I were a founder or CTO looking to bring in an AI implementation consultant this quarter, here is the exact sequence I would run:

1. Write a one-page scope with the workflow, the data source, the success metric, and the kill line. If you cannot write this, hire a consultant for a paid 1-week discovery instead of a build.
2. Talk to three candidates. Run the six technical questions above. Score them honestly.
3. Ask each for a fixed-scope 2-week PoC quote against your one-pager. Compare not just price but what they include (eval set, dashboard, docs).
4. Pick the one whose PoC plan includes an eval set and a kill criterion, even if they are not the cheapest. This is the single strongest predictor of a system that survives past month three.
5. Insist on weekly demos with real data. No slideware. If week 2 does not have a running system you can poke, something is wrong.
6. Plan the handover from day one. Docs, runbook, dashboards, and a working local dev environment are deliverables, not favors.

The consultants who ship are the ones who make the boring parts non-negotiable from the first call. That is the whole tell.

If you are scoping an AI implementation and want a second opinion on the plan (or on a proposal already on your desk), I am always happy to spend 30 minutes on it. You can reach me at lazar-milicevic.com/#contact, or dig through more of the production notes on the blog.