The Honest Numbers From My Last 5 AI Builds

Most AI automation case studies read like brochures. Big percentages, no denominators, no mention of what broke or what cost more than it saved. I keep a spreadsheet for every system I ship: token spend, infra cost, hours reclaimed, and whether the thing actually earned its keep after six months. Here is that spreadsheet, translated into English, across my last five builds. Two of them underperformed. I will tell you which and why.

The Five Builds At A Glance

Before the deep dive, the summary table. Numbers are monthly averages over the first six months in production, or the full life of the build if it was retired earlier. Dollar figures are USD. "Time saved" is measured against the pre-automation baseline the operator actually tracked, not a guess.

Two winners built around clear, repeated workflows. One infra win that paid for itself in software licenses. Two that did not earn their spot and taught me more than the wins did.

Build 1: The Multi-Agent Content Machine

This is BizFlowAI ContentStudio, the system that researches, writes, optimizes, and publishes across multiple sites without me in the loop. It has been running for over six months. The honest breakdown:

Monthly cost: roughly $180. That splits into ~$120 in LLM tokens (Claude for drafting and critique, a cheaper model for classification and metadata), ~$35 in serverless AWS (Lambda, EventBridge, S3, a small RDS instance), and ~$25 in third-party APIs for search data and rank tracking.

Time reclaimed: 45 hours per month against the manual baseline of researching, drafting, editing, formatting, publishing, and internal linking. I timed this over two months before automation. The number is not inflated.

Where the money actually goes: the critique agent. I run a generate-then-critique-then-revise loop, and the critique pass consumes roughly 35% of total tokens. I could cut it. I have not, because the 40% token reduction work I did earlier this year came from prompt compression and caching, not from removing the critique. Killing critique would save ~$40/mo and cost me an unknown amount in quality regressions. Not worth it.

What actually made it work: a self-learning loop that reads GSC data weekly, identifies which posts are gaining impressions, and biases the topic queue toward adjacent long-tails. That single feedback loop is why the system stopped producing vanity content and started producing content that gets impressions. Without it, this build would be in the "underperformed" pile.

Build 2: Serverless AWS + Zendesk For SLA Compliance

Smallest infra bill on the list, biggest strategic impact. The problem was simple: tickets were missing SLA because the routing logic lived in a human's head and that human went on vacation.

Monthly cost: ~$40. Lambda invocations, EventBridge rules, a tiny DynamoDB table for state, CloudWatch for observability. Zero LLM tokens. Yes, this counts as an "AI automation build" because part of the routing uses a classifier, but the classifier runs on cached embeddings and costs pennies.

Result: first-ever month of full SLA compliance for the team it served. That is the number the business cared about. The 18 hours per month of manual triage reclaimed was a side effect.

Lesson: the highest-ROI "AI" builds are often 90% boring event-driven plumbing and 10% intelligence. If you skip the plumbing and lean on the model to do everything, you get a demo, not a system. I wrote more about this in Agentic Workflows That Survive Real Inputs.

Build 3: Analytics Migration And Automated Reporting

This is the one that delivered EUR 30, 60k in annual savings (about $32k, 65k) by replacing a legacy analytics stack with a lighter, event-driven pipeline plus scheduled LLM-generated summaries for stakeholders.

Monthly cost: ~$95 all-in. Warehouse queries, a small orchestrator, and roughly $30 in tokens for the weekly and monthly narrative reports.

Where the savings came from: license consolidation, not the AI. The LLM narrative layer was the wedge that made stakeholders comfortable retiring the old tool ("but who will write the exec summary?"). Once that objection died, the migration paid for itself in the first quarter.

Honest caveat: if you subtract the license savings, the pure AI-generated-reporting value is maybe $400, 600/mo in analyst time. Real, but not the headline number. I try to keep those two things separate when I report to a client. Conflating them is how people lose trust in AI ROI claims.

Build 4: Local-LLM RAG That Underperformed

This one hurt because I liked the architecture. Ollama running a mid-size open model, pgvector for embeddings, hybrid search with reciprocal rank fusion, a clean retrieval API. Technically sound. I wrote the hybrid search post based on this exact build.

Monthly cost: ~$65 amortized (hardware plus power plus a small managed Postgres).

Time saved: only about 6 hours per month. Estimated value ~$480. Barely covers the cost when you include the maintenance I do on it.

Why it underperformed:

1. Query volume was too low. The internal team it served asked the system maybe 40, 60 questions a week. RAG shines at scale. At low volume, people just Slack a colleague.
2. The corpus was not the bottleneck. People were not blocked because they could not find docs. They were blocked because the docs did not exist for the questions they were asking. RAG cannot retrieve what was never written.
3. Trust cold-start. New users tried it twice, got one great answer and one mediocre one, and stopped. The mediocre answer was technically correct but poorly phrased. Retrieval quality was not the problem. Presentation was.

What I would do differently: I would validate with a two-week Slack-bot-with-a-human-in-the-loop before building the full retrieval stack. If the team is not asking questions to a human proxy, they will not ask a bot either. That is a $5 test that would have saved me a $65/mo permanent bill.

Build 5: The Outbound Qualifier I Killed At Month 4

An autonomous agent that ingested inbound leads, enriched them, ran a qualification prompt, and updated the CRM with a score plus a short reasoning paragraph. Sounds useful. Was not.

Monthly cost: ~$210. Higher than expected because enrichment API calls added up fast and the qualification prompt was long (roughly 3.5k input tokens per lead).

Time saved: ~4 hours per month. Value ~$320. Net negative.

Why it failed:

• The qualification signal it produced correlated with the sales rep's own gut about 75% of the time. On the 25% where they disagreed, the rep won 8 out of 10. The model was confidently wrong on edge cases that actually mattered.
• Volume was too low to justify the ceremony. Roughly 60, 90 leads a month. A human can score that in an hour with a spreadsheet.
• The "reasoning paragraph" the model produced became a liability. Reps started trusting it, and when it was wrong, they had already sent an email based on a bad framing.

I killed it at month 4. Refactored the useful parts (the enrichment layer and a much simpler rules-based scorer) into a 30-line script that costs $6/mo and does the job better. That script is still running.

Lesson: if a human can do the task in under an hour a month, do not build an agent for it. The floor cost of any agentic system, in tokens, tokens for retries, monitoring, evals, and your own attention, is higher than people admit.

What The Numbers Actually Say

Three patterns show up across all five:

1. Token cost is rarely the problem. Fit is. The build with the highest token spend (ContentStudio at $120/mo) is the biggest winner. The build with lower token spend (RAG at ~$20/mo in embeddings) underperformed. Fit to a real, repeated, high-volume workflow matters more than efficient prompting. Optimize prompts after you have proven fit, not before.

2. Boring infra beats clever prompts. The Zendesk SLA build has almost no LLM in it and produced the clearest strategic win. If you are pitching an AI project, ask yourself first whether the pain is actually an event routing problem, a data problem, or a workflow problem. If it is, solve that first and add intelligence where it genuinely helps.

3. The kill decision is the skill. Killing build 5 at month 4 was the highest-ROI decision in this whole spreadsheet. It stopped a $210/mo bleed, freed my attention, and produced a better $6/mo replacement. Most teams keep zombie AI systems alive because someone got promoted for launching them. Do not do that.

What I'd Do If I Were Starting Today

• Track the baseline before you build. Two weeks of honest time tracking. If you cannot show me the hours you spent on the task pre-automation, we cannot measure ROI post-automation.
• Set a kill threshold in writing. Mine is: if a build does not clear 3x its monthly cost in reclaimed time or hard savings by month 3, it goes on a "fix or kill" list. Month 4 is decision time.
• Instrument tokens per outcome, not tokens per call. "This agent costs $0.04 per completed useful task" is a number a CFO can price. "The model costs $3 per million tokens" is not.
• Prefer event-driven serverless for anything that runs unattended. Scale-to-zero economics change what is worth building. A build that costs $6/mo idle can sit there proving itself for a year without pressure.
• Publish the losers. If you only ever hear about the wins, you are being sold to.

Close

These are the actual numbers from actual systems, not slide decks. If you are weighing whether an AI automation build is worth it inside your own company, I am happy to talk through the shape of it honestly, including telling you when the answer is "do not build this." Reach out at lazar-milicevic.com/#contact, or read more on the blog if you want the deeper technical write-ups behind any of these builds.