A Week as an AI Integration Consultant

Most weeks I don't write much new code. I read other people's systems, draw arrows on a whiteboard, and try to figure out which of the twelve places customer data lives is the one I should actually trust. That is the honest shape of AI integration work in a B2B company that has been shipping since 2014. The LLM is the easy part. The plumbing is the job.

Here are field notes from a recent week doing exactly this, plus the checklist I wish every founder had in hand before they hire anyone (me included) to bolt an LLM onto their stack.

Monday: Mapping the data flow before touching a prompt

The client wanted "an AI assistant that answers customer questions from our knowledge base and CRM." That is a sentence, not a specification. My first day is almost always the same: I map where data actually lives, who writes to it, and how stale it is by the time anyone reads it.

For this client the map ended up looking like this:

The MySQL app is the one nobody wanted to talk about. It is also the only place that knows which features a given customer is actually entitled to. If the assistant answers "yes, you have access to X" without reading that database, it will hallucinate entitlements and create a support fire.

Rule I now apply on day one: find the ugliest system in the stack. That is almost always the real source of truth. The pretty SaaS on top is a view, not a fact.

I don't touch a prompt until I have this map signed off. If the CTO can't tell me which system wins in a conflict between Salesforce and NetSuite on the same field, we are not ready to add an LLM. We are ready to have a meeting about data governance dressed up as an AI project.

Tuesday: Picking the glue (queues, webhooks, or agents)

By Tuesday I know the systems. Now I have to pick the glue. This is where most integrations quietly fail six months in, because someone reached for the wrong tool on day one and everything after that inherited the mistake.

My rough decision tree, after doing this on and off for a decade:

• Synchronous HTTP call from the app to the LLM. Use when the user is waiting for the answer and latency budget is under two seconds. Chat UIs, autocomplete, inline summarization. Fails badly for anything that touches more than two backend systems.
• Webhook + queue (SQS, EventBridge, or a Postgres-backed queue). Use when work is triggered by an event (ticket created, invoice posted, form submitted) and the user does not need the answer in the same request. This is where 70% of real B2B integrations live. It is also where the money is, because it maps directly to "hours saved per month."
• Scheduled worker. Use for enrichment, backfills, nightly summaries, weekly digests. Cheap, boring, reliable. Underrated.
• Agent loop (multi-step, tool-using). Use only when the task genuinely requires branching decisions that a deterministic workflow cannot express. Most tasks people call "agentic" are actually a switch statement with anxiety.

For this client I went with an EventBridge bus in front of Lambda workers, with Zendesk and Salesforce webhooks feeding in, and one small agent loop for a specific case (drafting a reply that needs to check entitlements, then pull the right KB passage, then decide whether to escalate). Everything else is a linear workflow, because linear workflows are debuggable at 2am and agent loops are not.

The brittle trap I avoid: a prompt sitting inside a Zapier or Make step, calling an LLM, and writing back to a CRM with no queue, no retries, no idempotency key, and no audit log. It demos beautifully. It falls over the first time the LLM returns malformed JSON, and nobody can tell you which record got corrupted.

Wednesday: The retrieval layer nobody scopes for

Wednesday is always retrieval. The client had 4,200 Confluence pages, 18,000 closed Zendesk tickets, and a product manual as a 380-page PDF. "Just point the AI at it" is a six-month project disguised as a sentence.

What I actually built this week:

1. Ingestion workers that pull from each source on its own cadence (Confluence via API on a 6-hour schedule, Zendesk via webhook on ticket close, PDF once with a manual re-ingest hook).
2. A normalization step that strips boilerplate, chunks by semantic boundary (not fixed token count), and attaches metadata: source system, last modified, author, customer tier visibility.
3. Storage in Postgres with pgvector for embeddings and a tsvector column for full text search on the same rows.
4. Hybrid retrieval at query time using Reciprocal Rank Fusion to merge vector and lexical results. This alone fixed a class of failures where pure vector search kept missing exact product names and error codes.
5. A rerank pass with a small cross-encoder for the top 30 candidates before handing 6-8 to the LLM.

The number that matters here: on this client's evaluation set of 140 real historical questions, pure vector retrieval got 61% top-5 recall. Hybrid + rerank got 89%. That 28 point gap is the difference between "the assistant is useful" and "the assistant is a liability." It is also the reason I no longer take retrieval seriously if it is just a vector database and vibes.

Cost-side note: keeping retrieval in Postgres (instead of a dedicated vector DB) saved this client roughly $600 to $900 per month at their volume, and removed one vendor from the security review. For a mid-market B2B company that matters more than the theoretical benchmarks.

Thursday: Guardrails, evals, and the boring stuff that keeps you employed

Thursday is the day I earn my rate. Anyone can wire an LLM to a CRM. Making it not embarrass the company is the actual skill.

Three things I insist on before anything goes to production:

1. A structured output contract, enforced twice. The prompt asks for a specific JSON schema. The response is validated against a Pydantic or Zod schema before it touches a downstream system. On failure, the worker retries with an error-corrective prompt, up to two times, then drops to a dead-letter queue with the full trace. No silent failures, no half-written records.

2. An eval set that lives in the repo. For this client, 140 questions with expected behaviors, run on every prompt change and every model change. Not just "does it answer correctly," but categorical: did it refuse when it should have refused, did it cite the right document, did it escalate the entitlement question. I run this in CI. A prompt change with a 3-point regression on the eval set does not ship.

3. An audit log, always. Every LLM call gets stored with input, output, model, cost, latency, retrieval context IDs, and the outcome downstream. This costs almost nothing in Postgres and it is the single most valuable artifact you will have when a customer emails asking why the assistant told them something wrong three weeks ago.

The kind of thing that goes into a worker looks roughly like this:

def handle_ticket_event(event):
    ticket = fetch_ticket(event["ticket_id"])
    context = retrieve(ticket.body, customer_id=ticket.customer_id)
    entitlements = get_entitlements(ticket.customer_id)  # from MySQL, not CRM

    draft = llm.generate(
        prompt=REPLY_PROMPT,
        context=context,
        entitlements=entitlements,
        schema=DraftReply,  # enforced
    )

    audit.log(event, context, draft)  # always, before any write

    if draft.confidence < 0.7 or draft.needs_human:
        assign_to_agent(ticket.id, draft=draft.text)
    else:
        post_internal_note(ticket.id, draft.text)  # never auto-send week 1

Note the last comment. Week one, the assistant never sends anything to a customer. It writes internal notes for agents to review. Week four, once the eval set and the audit log show it is behaving, we flip the switch on the low-risk categories. This is how you ship an LLM into a real business without a Slack channel full of angry executives on day two.

Friday: Handover and the parts I refuse to skip

Friday is documentation and handover. This is where independent consultants lose clients (by writing nothing) and where I keep them (by writing too much, honestly).

What I hand over on every engagement:

• An architecture diagram that matches what actually got built, not the one from the proposal.
• A runbook for the top 8 failure modes: LLM API down, retrieval empty, schema validation failing, webhook backlog, quota hit, downstream write failure, silent degradation, customer complaint triage.
• The eval set, with instructions to add to it every time something goes wrong in production. The eval set is a living asset. It is arguably the most valuable artifact of the entire project.
• Cost dashboards per workflow, so someone can answer "is this saving us money" without asking me.
• A 30-day, 60-day, and 90-day review checklist.

Clients who run the 30-60-90 reviews get compounding value. Clients who don't tend to let the whole thing quietly rot within a year. I now write this into the contract.

The checklist I wish founders used before hiring anyone

If you are a founder or CTO thinking about hiring an AI integration consultant, run through this before you write the brief. It will save you and your consultant a painful week.

1. Name the outcome in hours or dollars. "Reduce first-response time on tier-2 tickets by 40%" or "eliminate 20 hours per week of manual invoice reconciliation." Not "add AI."
2. List every system that holds the data the LLM will need. Include the ugly ones. Especially the ugly ones.
3. Identify the source of truth for each conflicting field. If two systems disagree on customer status, which wins? Write it down.
4. Decide the failure mode. When the LLM is wrong, what happens? Does it write to a customer, or draft for review? Week one should always be draft for review.
5. Budget for retrieval and evals, not just prompts. If the proposal is 80% "prompt engineering" and 20% everything else, the ratio is backwards.
6. Ask who owns the audit log. If nobody, that's the first thing to fix.
7. Confirm you have 100+ real historical examples the assistant would have handled. If you don't, the first two weeks are collecting them, not building.
8. Agree on the review cadence. 30, 60, 90 days. In writing.
9. Ask the consultant what they will not do. If the answer is "anything," walk away.

What I'd do if I were starting fresh today

Start with the smallest workflow that touches real money or real customer time. One ticket type. One report. One approval step. Wire it end to end with a queue, a schema, an eval set, and an audit log. Ship it as an internal draft first, promote it to customer-facing only when the numbers say so. Then, and only then, look at the next workflow.

The teams that get AI integration wrong try to boil the ocean with a chat interface. The teams that get it right pick one boring workflow, instrument it obsessively, and let the ROI compound. Everything I have shipped that survived past a year followed the second pattern.

If you are wrestling with an integration like this and want a second pair of eyes, or you have already tried the prompt-in-a-Zap route and want to do it properly, get in touch at lazar-milicevic.com/#contact. More field notes from production systems live on the blog.