How to Choose an AI Automation Agency (SMB Guide)

Last month a founder sent me a proposal from an "AI automation agency" quoting $48,000 for what turned out to be three Zapier zaps and a GPT prompt wrapped in a Streamlit UI. He asked if the price was fair. It was not. But the harder question is the one he did not know to ask: would any of it still be running in six months, and who owns it when the agency disappears?

If you run a small business and you are about to hire someone to automate your ops, this is the checklist I wish every founder had before signing. It is the same lens I use when small business owners hire my team at BizFlowAI, and the same one I apply to my own subcontractors.

Start with the workflow, not the AI

Before you evaluate any agency, write down the workflow you want automated in plain language, with numbers. Not "automate customer support." Instead: "We get about 140 support emails a week, 60% are refund requests, 25% are shipping status, 15% are everything else. A human currently spends 12 hours a week on this and response time is 9 hours." That paragraph is worth more than any agency deck.

Any agency that starts the conversation with "we use GPT-4o and LangGraph" before asking what your week actually looks like is selling technology, not outcomes. The good ones will spend the first call asking:

• What triggers the work today, and who does it?
• Where does the data live (email, spreadsheet, CRM, database)?
• What does "done" look like, and who verifies it?
• What is the cost of a wrong answer? (A misrouted ticket is cheap. A wrong refund is not.)
• What breaks in your business if this automation goes down for 48 hours?

That last question separates real engineers from prompt tinkerers. If they have no answer, they have never shipped a system that runs unattended.

The three pricing models, and which one to avoid

There are basically three ways an AI automation agency will price a project. I have used all three and I will tell you honestly where each one is right and wrong.

My honest opinion: for a first project with a new agency, insist on a fixed-fee discovery sprint (usually 2 to 4 weeks, $5k to $15k) that produces a scoped build proposal, a working prototype of the riskiest piece, and a written architecture. Then decide whether to continue with a fixed-price build or a capped T&M engagement.

Avoid pure monthly retainers until you have shipped one project together and you actually know what you are buying each month. "$4,000/month for AI automation services" with no defined deliverables is how founders end up paying $48k a year for a Slack bot.

Tech stack red flags (and green flags)

You do not need to be technical to spot most of these. Ask the agency to draw their proposed architecture on a whiteboard or a Miro board during the sales call. If they cannot, that is your first answer.

Red flags:

• Everything runs in n8n or Make.com with no code fallback. These tools are fine for glue, but if the entire business logic lives in a visual workflow builder, you cannot version-control it, test it, or migrate it. You are locked in.
• "We use our own proprietary platform." Translation: when you leave, you lose everything. Ask if you get the code and can run it on your own infrastructure.
• No mention of evals or monitoring. If they cannot tell you how they will measure whether the LLM is giving correct answers in production, they will not know when it starts failing. And it will start failing.
• A single-model dependency with no abstraction. "We use GPT-4o for everything" is a business risk. Prices change, models get deprecated, latency shifts. Good builds route between Claude, GPT, and a local model (Ollama, Llama) depending on the task.
• No retry logic, no dead-letter queue, no idempotency. Ask them: "What happens if the OpenAI API times out at 2am on a Sunday?" A blank stare means you are the QA team.

Green flags:

• Code lives in your GitHub org from day one.
• Infrastructure as code (Terraform, SST, CDK, or at least documented click-ops).
• A staging environment that mirrors production.
• Written evals: a set of input/output test cases the LLM output is graded against on every deploy.
• Observability: logs, traces, cost dashboards. I use Langfuse or a simple Postgres + Grafana setup for most SMB projects.
• Serverless-first architecture (AWS Lambda, EventBridge, API Gateway, or the equivalent on GCP/Azure) so you pay for what you use and it scales to zero when idle.

On the serverless point specifically: a small business does not need a $600/month EC2 cluster to run 2,000 automations a week. A well-designed event-driven system on AWS Lambda often costs under $40/month at that volume, plus LLM API costs. If your agency proposes a "24/7 always-on server" for a workload that runs in bursts, they are either lazy or they mark up hosting.

The automations that actually move the needle for SMBs

I have built dozens of these. The pattern is depressingly consistent: the automations that get pitched in agency decks (AI chatbots, AI content, AI voice agents) are rarely the ones that produce the biggest ROI. The boring ones do.

Here is what I consistently see deliver real hours-saved and dollars-saved for small businesses:

1. Inbound triage and routing. Email, form submissions, or ticket classification with a confidence threshold. Above the threshold, auto-route or auto-respond. Below, escalate to a human with a summary and suggested reply. On one integration I built with AWS and Zendesk, this pattern took a support team from missing every SLA to hitting them for the first time.
2. Document extraction to structured data. Invoices, contracts, PDFs, receipts. Turning unstructured PDFs into rows in a database is a well-solved problem with modern LLMs and it consistently saves 8 to 20 hours a week per person doing manual data entry.
3. Internal RAG over company docs. A retrieval system over your SOPs, contracts, pricing sheets, and past project notes, hit through Slack or a simple web UI. I typically use pgvector with hybrid search (BM25 + vector, fused with RRF) because it is cheap, portable, and gives better precision than pure vector search. Employees stop asking each other the same questions.
4. Scheduled research and report generation. Competitor pricing, industry news digests, weekly ops reports. A cron-triggered agent pipeline that runs while you sleep and lands a formatted doc in your inbox at 7am.
5. CRM enrichment and lead scoring. Pulling public data on new leads, scoring them against your ICP, writing a personalized first-line for outreach.

Notice what is not on this list: customer-facing AI chatbots. They can work, but the failure cost is high (angry customers, brand damage) and the "wow" moment for the founder rarely translates to measurable dollars. I usually push clients to automate the back office first, prove the ROI, then move to customer-facing work with the confidence and eval infrastructure you built along the way.

For context, in a recent 4-system automation ecosystem I built for a small business, we tracked 73+ hours saved per month across the team and a 192% Year-1 ROI. None of those four systems was a chatbot. All four were boring internal workflows.

Ownership, exit, and the "hit by a bus" test

Before you sign, get written answers to these five questions:

1. Who owns the code? The answer must be you. Not "we license it to you." You. In your GitHub org, from commit one.
2. Who owns the data? You. Including embeddings, fine-tuning datasets, and eval sets.
3. Where do the API keys live? In your accounts (OpenAI, Anthropic, AWS), billed to your card. Not in the agency's account with a markup.
4. Can we hire another engineer to take this over tomorrow? Ask for the README they would hand a new engineer. If it does not exist, they have not written one, and you are the documentation.
5. What is the offboarding process? A good agency has one. A bad agency gets defensive.

I run every project on the assumption that the client should be able to fire me next Tuesday and keep everything running. That is not a weakness, it is a feature. It forces good engineering, and it is the single strongest signal of whether you are working with a professional or a hostage-taker.

The founder's checklist (print this)

Use this on your next agency call. If they cannot answer most of it clearly, keep looking.

Scoping:

• [ ] Did they ask about the current workflow and cost of errors before pitching tech?
• [ ] Did they propose a discovery sprint with a fixed deliverable?
• [ ] Is the success metric in hours saved or dollars saved, not "AI implemented"?

Pricing:

• [ ] Is the first engagement fixed-fee, not open-ended retainer?
• [ ] Are LLM API costs billed to your account, not marked up through theirs?
• [ ] Is there a written change-order process?

Tech:

• [ ] Code in your GitHub org from day one?
• [ ] Serverless or scales-to-zero architecture where appropriate?
• [ ] Written evals for any LLM-generated output?
• [ ] Monitoring, logging, cost dashboards from day one?
• [ ] Model-agnostic (can swap Claude, GPT, local) without a rewrite?

Ownership:

• [ ] You own the code, data, keys, and infrastructure?
• [ ] README exists for a hypothetical new engineer to take over?
• [ ] Offboarding process is written into the contract?

Team:

• [ ] Can you talk directly to the engineer who will build it, not just an account manager?
• [ ] Do they have shipped case studies with real numbers (hours saved, dollars saved, SLA metrics)?
• [ ] Are they honest about what AI is bad at, not just what it is good at?

What I'd do if I were hiring an agency tomorrow

Pick the smallest workflow with the biggest current pain. Spend $8k to $12k on a fixed-fee discovery sprint with two agencies in parallel if you can afford it. Ask each to ship a working prototype of the riskiest technical piece (not a slide deck) in 3 weeks. Whichever team writes cleaner code, communicates better, and is more honest about trade-offs wins the build.

Do not fall for the agency with the fanciest branding. Fall for the one whose engineer answers your Slack message on a Saturday with "we tried that pattern, it broke in production for this exact reason, here is what we do instead." That is the tell.

And measure everything from day one. Hours saved per week, cost per automation run, error rate, escalation rate. If your agency is not putting these numbers in front of you monthly, they are asking you to trust vibes. Do not.

---

If you want a second opinion on a proposal you have received, or you want to talk through a specific automation before you scope it out, get in touch at lazar-milicevic.com/#contact. I keep more notes like this on the blog, including how I run PoCs that actually ship and how I scope automation work in 30 days.