5 Production AI Agent Systems I Run and What They Do

Most "agent" demos fail at the same point: the second night. The first run works, you screenshot it, and then something quietly drifts while nobody is watching. I build systems meant to run unattended, so my definition of "working" is boring: it runs, it fails safely, and I find out about it. Here are five systems I actually operate, what each one owns, and where I still keep a human in the loop.

One note on numbers. I'm not going to invent reliability percentages or per-run costs for you. Where I give a figure, it's a verified fact from the sources linked below or a label like "example." The honest part of this post is the structure of each system, not a vanity metric.

The test I apply before calling anything an agent

Anthropic draws a useful line between workflows (LLMs and tools orchestrated through predefined code paths) and agents (the model directs its own process), and recommends starting with the simplest solution that works (Building effective AI agents). They also say plainly that agentic systems trade latency and cost for better task performance, so you should weigh that before building one.

I take that literally. Of the five systems below, only some are true agents. The rest are workflows with an LLM step in them, and I'm happy with that. A deterministic pipeline with one well-scoped model call beats a free-roaming agent almost every time I can get away with it. I reach for real agency only where the path genuinely can't be known in advance.

1. The content research-to-publish pipeline

The task it owns: take a topic target, research it, draft an article, optimize it for search and answer engines, and publish it to a site without me touching it.

Tools: a research step that pulls sources, a drafting step, a structure and SEO/AEO pass, and a publisher that writes to the CMS. This is the core of BizFlowAI ContentStudio, which runs across multiple sites.

Guardrails: this is where I learned the most. The drafting model is never trusted to decide facts. Statistics and links come from a verified research block that is passed in as data, and the writer is instructed to describe, not recall, anything outside it. A separate check rejects drafts with claims that don't trace back to that block. The publisher is idempotent: it keys on a slug so a retry can't double-post.

Failure modes:

• The model writes a confident number from memory. This is the big one, and the research-block constraint exists because of it.
• The same topic gets generated twice under slightly different titles. I catch this with a distinct-intent check against recent titles before generation starts, not after.
• A publish succeeds but the downstream index call fails, leaving a live page nobody can discover.

Human oversight still needed: I review a sample, not everything, and I read the failures in full. The pipeline is autonomous in the sense that it doesn't wait for me. It is not autonomous in the sense that I stopped looking.

2. The self-learning content loop

The task it owns: decide what to write next from real search performance, then feed that back into the pipeline above.

The loop is five verbs in order: measure, learn, target, generate, publish. It reads search performance data, works out which pages have exposure but weak position, and turns that into the next batch of targets. Pages already ranking in the middle of the results with decent impressions get refreshed before brand-new topics get commissioned.

Why it's more workflow than agent: the control flow is fixed. The model contributes judgment at specific steps (interpreting which queries signal real intent, naming a sharper title), but the loop itself is deterministic code. I tried a looser version where a model decided the whole plan. It was more impressive in a demo and worse in operation, because it would chase novelty instead of fixing the unglamorous page that was one good edit from page one.

Failure modes:

• Optimizing on thin data. A handful of clicks over a month is noise, and an eager loop will "learn" a pattern from it. I treat low-volume signals as tentative and weight them accordingly.
• Feedback loops that reinforce themselves. If the loop only ever targets what already worked, it narrows. I force a share of exploration so it doesn't collapse into one topic.

Human oversight: I set the guardrails on how aggressively it can act on weak signals. That policy is mine. The loop executes it.

3. The serverless AWS + Zendesk SLA integration

The task it owns: keep ticket handling inside its service-level commitments by reacting to ticket events automatically. This integration delivered the first-ever SLA compliance for the team that used it.

Tools: Lambda functions triggered by events, EventBridge for scheduling and routing, API Gateway as the entry point, and the Zendesk API as the system of record.

This one is a good reminder that not everything "agentic" needs a model in the hot path. The reliable core is event-driven and deterministic. The constraints that shaped it are concrete, and they come straight from AWS:

Guardrails: because a function can be invoked more than once for the same event, every handler is written to be safe to repeat. Ticket updates carry an idempotency key. Anything that could exceed a few seconds is split so no single invocation depends on beating the timeout.

Failure modes:

• Duplicate events causing duplicate ticket actions (solved by idempotency).
• A large payload silently exceeding the async limit and being dropped.
• An upstream API slowing down and turning a 3-second default into a stream of timeouts. Setting timeouts deliberately, per function, matters more than people think.

Human oversight: the system escalates to a person on anything ambiguous. It enforces policy; it doesn't invent exceptions to it.

If you need a step to wait on a person for a long time, standard Lambda isn't the tool. AWS notes that durable functions can run a durable execution for up to one year when invoked asynchronously, for example for human-approval workflows (AWS Lambda FAQs). Worth knowing before you build your own waiting logic.

4. The local-LLM RAG assistant

The task it owns: answer questions over a private document set without that data leaving the machine.

Tools: a local model served through Ollama, embeddings stored in Postgres with pgvector, and hybrid retrieval that merges keyword and vector results with reciprocal rank fusion (RRF).

Why hybrid: pure vector search is good at meaning and bad at exact strings. Part numbers, error codes and names are exactly the things people search for, and embeddings blur them. Keyword search nails those and misses paraphrase. Fusing the two ranked lists with RRF gets you both without having to hand-tune score weights across incompatible scales.

Guardrails: the assistant is instructed to answer only from retrieved passages and to say so when retrieval comes back empty or weak. The fix for hallucination here is rarely a better prompt. It's a retrieval step good enough that "I don't have that" is the correct answer more often than a guess.

Failure modes:

• Chunking that splits the answer across two chunks so neither retrieves well.
• Stale embeddings after documents change. Re-embedding on change is boring and essential.
• A small local model that follows instructions less reliably than a frontier API model. That's a real trade-off you accept for privacy and cost, and I size tasks to match.

Human oversight: I spot-check answers against sources, especially after any change to chunking or the embedding model, because those changes shift behavior invisibly.

5. The small-business automation ecosystem

The task it owns: replace recurring manual operations for small businesses with systems that run on their own. The four-system ecosystem I built saved 73+ hours a month and returned 192% in year one.

I'm keeping the specifics of those systems general on purpose, but the lesson transfers: the ROI didn't come from the cleverest model. It came from picking operations that were repetitive, rule-shaped and tedious, and automating those first. Hours saved is a number a business owner can verify, which is why I anchor every engagement to it.

Failure modes:

• Automating a broken process. You just get the wrong result faster.
• No owner on the client side. Unattended doesn't mean unowned; someone has to care when it alerts.

Human oversight: every one of these has an alert path and a named person who receives it.

What I'd do

If you're building your first production agent, here is what I'd actually do, in order.

1. Default to a workflow. Make the control flow code. Add model judgment only at the steps that need it. Promote a step to true agency only when you can't enumerate the path.
2. Spend your time on tools, not the prompt. Anthropic reports that building its SWE-bench agent, they spent more time optimizing tools than the overall prompt, and that requiring absolute file paths fixed the model's path mistakes (source). That matches my experience: constrain the tool interface and a whole class of errors disappears.
3. Make every action idempotent. If a retry can double-charge, double-post or double-notify, you will eventually find out the hard way.
4. Pass facts in; don't ask the model to remember them. Hallucinated numbers are the most common production defect I see, and the cure is architectural.
5. Build the alert before the feature. An agent you can't observe is an agent you can't trust.

On the risk side, the sober view is worth hearing. Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls (Gartner). That's a forecast, not an observed rate, but the three causes line up with the failures I see: cost nobody modeled, value nobody defined, and controls added too late. Gartner also estimates that only about 130 of the thousands of agentic AI vendors are real, calling the rest "agent washing." Treat vendor claims accordingly.

One more thing on oversight: the EU AI Act's Article 14 requires high-risk AI systems to be designed so people can effectively oversee them (Article 14). That obligation applies only to systems classified as high-risk. In my own assessment, the systems I describe here don't fall into the high-risk categories listed in Annex III, but that is my reading, not legal advice, so check the European Commission's AI Act overview or a qualified lawyer for your own case. I keep humans in the loop anyway, because it's good engineering rather than because a regulation demands it.

If you're weighing an agent project and want a second opinion on whether it should be an agent at all, I'm happy to talk it through at lazar-milicevic.com/#contact. Or keep reading on the blog, where I write up the systems that work and the ones that didn't.