LLM Engineering Consulting: Building AI That Runs Unattended

The most expensive AI bug I have ever shipped was not a hallucination. It was a scheduled worker that retried a failing step in a loop at 3 a.m., with nobody watching, and burned through far more tokens than it should have before anyone noticed. The model was fine. The system around it had no opinion about failure. That gap is what LLM engineering consulting is really about: not picking a model, but making the whole thing behave when no human is in the loop.

After 10+ years of building AI automation and cloud systems, and running my own autonomous pipelines under BizFlowAI, this is the playbook I follow when someone asks me to make an AI feature reliable.

The scope: what an LLM engineering consultant actually owns

An LLM engineering engagement covers six things: architecture, evals, retrieval, agents, cost control and production hardening. Prompt writing is a small slice of it, and usually the least valuable part.

Here is how I break it down when I scope work:

The order matters. Most teams come to me wanting to talk about agents. I almost always start at architecture and evals, because the answers there decide whether the rest is worth building.

Start with the least autonomy that works

The best architecture decision is usually to build less. Anthropic's guidance on building agents says to find the simplest solution possible and only increase complexity when needed, which can mean not building an agentic system at all. It also notes that for many applications, optimizing a single LLM call with retrieval and in-context examples is enough. Source: Building effective AI agents.

I agree with that from the field, and I would add the reason: agentic systems trade latency and cost for task performance, and you inherit every failure mode of a loop. So my default ladder looks like this:

1. A single call with good context and examples.
2. A fixed workflow: several calls in a predetermined order, each step testable on its own.
3. A workflow with routing, where a cheap model classifies and hands off.
4. A true agent that decides its own next step, only where the path genuinely cannot be known in advance.

Most of the systems I run unattended live at levels 2 and 3. My content pipeline is a good example: it moves through stages of research, writing, optimization and publishing. The "autonomy" is in the scheduling and the self-learning loop (measure, learn, target, generate, publish), not in a model freely improvising. That is what lets it run across multiple sites without me watching.

When a client wants an agent, I ask one question: could a flowchart do this? If yes, we draw the flowchart.

Evals: the part that decides whether you can change anything

Without evals, every prompt edit and model upgrade is a gamble. This is the single highest-leverage thing a consultant installs, and it is the thing most teams are missing when I arrive.

My minimal setup, in the order I build it:

1. Collect real cases. Pull 30 to 50 real inputs from logs or the support queue, including the ugly ones. Synthetic data comes later.
2. Define pass/fail per case, not a vibe score. "Extracted the correct invoice total" beats "good quality."
3. Separate deterministic checks from judged checks. Schema validation, required fields, forbidden content: code. Tone, completeness, faithfulness: an LLM judge, calibrated against a handful of human-labeled examples.
4. Run on every change. Prompt edit, model swap, retrieval tweak, all of it. A failing eval blocks the deploy.
5. Feed production failures back in. Every real bug becomes a permanent test case.

A short sketch of the shape (not a framework, just the idea):

def run_suite(cases, system):
    results = []
    for c in cases:
        out = system(c.input)
        results.append({
            "id": c.id,
            "schema_ok": validate_schema(out),
            "checks_ok": all(chk(out, c) for chk in c.deterministic_checks),
            "judge_ok": judge(out, c.rubric) if c.rubric else True,
        })
    return results  # gate the deploy on the failure rate

One trap worth naming: passing evals do not mean the agent works for customers. If your eval set only contains the cases you thought of, you are measuring your imagination. Production traffic is the real test set, which is why step 5 is not optional.

Retrieval: most RAG failures are not model failures

When a RAG system gives a wrong answer, the retriever is the suspect more often than the model. The model dutifully reasoned over the wrong context. I have built local-LLM RAG pipelines with pgvector and hybrid search using reciprocal rank fusion (RRF), and the lessons repeat.

What I check, in order:

• Chunking against the real query shape. Chunks sized for how documents are written, not how questions are asked.
• Hybrid over pure vector. Dense embeddings miss exact identifiers, SKUs, error codes and names. Keyword search plus vector search fused with RRF fixes a surprising number of "it cannot find the obvious thing" complaints.
• Measure retrieval separately from generation. Before you ever look at the final answer, ask: was the right passage in the top-k at all? If not, no prompt will save you.
• Metadata filters. Tenant, date, document type. A correct answer from the wrong customer's data is a security incident, not a quality issue.
• Let it say "not found." A system that must always answer will invent. Build the refusal path and test it.

Splitting retrieval evals from end-to-end evals is the single change that most speeds up debugging. It turns "the bot is bad" into "recall at 5 is low on contract documents," which is a problem you can actually fix.

Cost control and the guardrails that make unattended runs safe

Unattended systems need hard limits, because nobody is there to notice the bill. The OWASP Top 10 for LLM Applications (2025 edition) names this directly: LLM10:2025 Unbounded Consumption, alongside LLM06:2025 Excessive Agency and LLM01:2025 Prompt Injection. Source: OWASP Top 10 for LLM Applications 2025. Those three map almost exactly onto what I harden in every engagement.

On the safety side, OWASP breaks Excessive Agency into excessive functionality, excessive permissions and excessive autonomy, meaning high-impact actions with no human in the loop (summary). My practical translation:

• Give each agent the minimum tool set for its job, nothing more.
• Scope credentials per task, read-only wherever possible.
• Put a human approval gate in front of anything irreversible (sending, deleting, paying).
• Cap iterations, tokens and wall-clock time per task, and fail closed when a cap trips.

On the cost side, the levers are real but the numbers move fast, so check current pricing on the vendor's own page before you plan around it. Third-party trackers currently report that Anthropic's Batch API gives a 50% discount on asynchronous jobs, and that this stacks with prompt-cache pricing (BenchLM). Cache-read discounts differ by model, so I never plan around a single flat "caching saves X%" figure. What I do instead:

• Route by difficulty. A small, cheap model handles classification and extraction. The expensive model only sees what needs it.
• Cache stable prefixes. System prompts and reference material that repeat across calls belong in the cache.
• Batch whatever is not latency-sensitive. Overnight jobs are the perfect fit.
• Track cost per successful task, not cost per call. A cheap model that needs three retries can cost more than a strong one that gets it right once.

Here is a deliberately round, illustrative example (not a real client figure): if a task costs $0.02 and succeeds 90% of the time, your real cost per success is about $0.022. If a cheaper variant costs $0.01 but succeeds 50% of the time, it is $0.02 per success and quietly burns your quality. Always divide by success rate.

Production hardening: boring engineering wins

The difference between a demo and a system that runs for months is almost entirely ordinary reliability engineering. My background in serverless AWS (Lambda, EventBridge, API Gateway) shapes this. The AWS and Zendesk integration I built delivered first-ever SLA compliance for that team, and not because of anything clever with a model. It came from retries, idempotency and alerting being treated as first-class.

The checklist I carry into every project:

• Idempotent steps. Re-running a failed job must not double-send or double-charge.
• Bounded retries with backoff, then a dead-letter queue and an alert. Never an infinite loop (see: my 3 a.m. story).
• Structured outputs and validation on every model boundary. Treat model output as untrusted input.
• Tracing per run. For any output, I can reconstruct the prompt, retrieved context, tool calls and cost.
• Model and prompt versioning. Pin versions, and roll forward through the eval gate, never by surprise.
• Heartbeat monitoring. A scheduled worker that silently stops is worse than one that errors loudly. Alert on absence.
• Graceful degradation. If the model API is down, the system queues work or falls back, rather than failing the user.

If you build on tool protocols, watch their lifecycle too. The Model Context Protocol released a 2026-07-28 specification with a stateless core, and it deprecated several features (Roots, Sampling, Logging, Dynamic Client Registration and the legacy HTTP+SSE transport) under a policy that keeps deprecated features available for at least 12 months (MCP announcement). That is the kind of change a pinned, eval-gated pipeline absorbs calmly, and a hand-wired one discovers in production.

If you ship into the EU, compliance belongs in the scope as well. Under the Digital Omnibus, Regulation (EU) 2026/1744, most Article 50 transparency duties such as chatbot disclosure already apply from 2 August 2026, while stand-alone high-risk obligations moved to 2 December 2027 (Orrick, technology.org). Whether any of it applies to you depends on the use case, so get legal review rather than trusting a blog post, including this one.

What I'd do

If I were handed an AI feature that needs to run unattended, this is the order I would work in:

1. Write down the task and the failure cost. What is the worst thing this can do wrong, and who is hurt?
2. Pick the lowest rung of the autonomy ladder that meets the requirement. Default to a workflow, not an agent.
3. Build the eval set before the second prompt iteration. Real cases, pass/fail, gated deploys.
4. Measure retrieval on its own if there is any RAG in the path.
5. Set the caps first: tokens, iterations, time, permissions, spend per task. Then relax them deliberately, never the other way around.
6. Add tracing and absence alerts before launch, not after the first incident.
7. Review cost per successful task monthly, and re-route to cheaper models when the evals say it is safe.

None of this is glamorous. All of it is what lets a system run while you sleep, and I would take that over a clever demo every time.

If you are weighing an AI project and want a second pair of eyes on the architecture, evals or cost model, you can reach me at lazar-milicevic.com/#contact. Otherwise, the rest of the blog goes deeper on agents, RAG and production hardening.