How to Build AI Agents From Scratch: A Blueprint

Every framework I have used in production, I have eventually gutted. LangChain, CrewAI, even parts of LangGraph. Not because they are bad, but because when an agent fails at 3 a.m. and you are staring at a stack trace inside three layers of abstractions you did not write, you start thinking: this loop is maybe 200 lines. Why do I need 40 dependencies to run it?

This is the architecture I actually use to ship autonomous multi-agent systems, the ones that publish content, hit APIs, and run unattended for weeks. No framework religion. Just the parts that survive real traffic.

The agent loop is smaller than you think

An agent is a while loop around an LLM call that can invoke tools and decide when to stop. That is the whole thing. Everything else (memory, planning, reflection, multi-agent choreography) is a variation on that core.

Here is the minimal loop I start every new agent from. Python, no framework, uses the Anthropic SDK but the shape is identical for OpenAI:

def run_agent(task: str, tools: dict, max_steps: int = 20):
    messages = [{"role": "user", "content": task}]
    for step in range(max_steps):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            system=SYSTEM_PROMPT,
            tools=[t.schema for t in tools.values()],
            messages=messages,
            max_tokens=4096,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason == "end_turn":
            return extract_text(response)

        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                result = tools[block.name].run(**block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": str(result),
                    "is_error": isinstance(result, Exception),
                })
        messages.append({"role": "user", "content": tool_results})

    raise MaxStepsExceeded(f"Agent did not finish in {max_steps} steps")

That is roughly 30 lines and it will run a real agent. Everything I am about to describe is layered on top of this without changing the shape of the loop.

Two things I have learned to bake in from day one: a hard step cap, and structured tool_result blocks with an is_error flag. Without the step cap, one bad prompt burns $40 in a night. Without the error flag, the model quietly hallucinates that failed calls succeeded.

Tool design is where agents actually live or die

The model is only as smart as the tools you give it. In production, 80% of my debugging time is not prompt tuning, it is tool interface work. Ambiguous tool names, missing error messages, and untyped inputs cause more agent failures than any model weakness.

The rules I follow, without exception:

1. Every tool is a pure function with a typed schema. No hidden state. If the tool needs context, pass it in.
2. Tool names are verbs. search_docs, not docs_util. The model prompts itself with the name.
3. Errors are returned, not raised. Every tool returns {"ok": bool, "data": ..., "error": ...}. The model can then react.
4. Descriptions include when NOT to use the tool. This alone cut wrong-tool-selection rates by half in one of my content agents.
5. Idempotency where possible. Retries are cheap when the tool is safe to call twice.

Here is a tool description snippet I have shipped:

{
  "name": "publish_post",
  "description": "Publish a finalized post to the CMS. Only call this AFTER `validate_post` returns ok=true. Do NOT call this to preview or draft. This action is irreversible for 5 minutes.",
  "input_schema": { ... }
}

That last line, "irreversible for 5 minutes", changes model behavior meaningfully. Claude and GPT-4-class models actually respect it. They pause and often call validate_post first, even when I have not explicitly told them to in the system prompt.

Memory: three layers, not one blob

The most common mistake I see in home-rolled agents is treating memory as "just append to the context window". That works for 5 turns. It falls apart at 50, and it becomes unaffordable at 500.

I split memory into three layers, each with its own retention policy:

In one of my recent posts I broke down how I cut a 3.26M token conversation down to 118K by moving from a naive append-everything design to this layered model. The key insight: the model does not need history, it needs the right slice of history.

For semantic memory I use pgvector with hybrid search (BM25 + vector, fused with RRF). Pure vector search misses exact-match queries ("what is the SKU for product X"), pure keyword misses semantic queries ("similar customer complaints"). Hybrid wins in every eval I have ever run.

For working memory, I aggressively summarize. Every 10 turns, a small model (Haiku or GPT-4o-mini) compresses the last 10 messages into 200 tokens of "here is what happened and what state we are in". The full log stays in the database for debugging. The context stays lean.

Orchestration: single agent, then supervisor, then graph

Do not start with a multi-agent system. I know it is the fashionable thing. Start with one agent and a lot of tools. Escalate only when you hit a real limit.

Here is the progression I actually use:

Level 1: Single agent, many tools. Works up to about 15-20 tools before the model starts confusing them. Cheap, easy to debug, testable.

Level 2: Supervisor plus workers. A planning agent decides which specialist to call. Specialists are themselves single agents with narrower tool sets. This is how BizFlowAI ContentStudio is built: a research agent, a writing agent, an SEO agent, a publishing agent, orchestrated by a supervisor that owns the workflow state.

Level 3: Graph / DAG. Only when you have deterministic branching, parallel work, and cycles that would be wasteful to let an LLM figure out on its own. This is where LangGraph earns its keep, and honestly, if I am at this level, I sometimes reach for it rather than rolling my own state machine.

The heuristic: if you can draw the workflow on a napkin and it does not need to change per input, the orchestration should be code, not an LLM decision. Every decision you outsource to the model is latency, cost, and a failure mode. Reserve LLM decisions for genuinely ambiguous branches.

Failure handling is 40% of production agents

The demo works. The production version has to survive rate limits, tool timeouts, malformed JSON, hallucinated tool names, infinite loops, context overflow, and the occasional model outage. In my experience, roughly 40% of the code in a production agent is failure handling, not "agent logic".

The layers I always put in:

• Retry with backoff on transient errors. Model APIs 529, tool network calls, everything.
• Structured output validation. If a tool expects JSON, validate it against a Pydantic schema and hand the model back the validation error on failure. The model will fix it in one turn 90% of the time.
• Loop detection. I hash the last three tool calls plus arguments. If the same hash repeats, I inject a system message: "You just repeated the same action. Reconsider your approach." That alone kills most infinite loops.
• Budget caps per run. Max steps, max tokens, max wall clock, max dollar. Any breach halts the run and logs it.
• Observability from day one. Every step is written to a trace store with inputs, outputs, latency, cost. I use a lightweight Postgres schema for this, not a hosted tool, so I can query traces with SQL. When something breaks at scale, you need to be able to ask "show me every run where the SEO agent called publish_post before validate_post".

Here is the pattern I use for tool retries:

def run_tool_safely(tool, args, max_retries=2):
    for attempt in range(max_retries + 1):
        try:
            result = tool.run(**args)
            return {"ok": True, "data": result}
        except TransientError as e:
            if attempt == max_retries:
                return {"ok": False, "error": f"Transient error after {max_retries} retries: {e}"}
            time.sleep(2 ** attempt)
        except ValidationError as e:
            return {"ok": False, "error": f"Bad arguments: {e}. Please correct and retry."}
        except Exception as e:
            return {"ok": False, "error": f"Tool failed: {e}"}

Notice the model sees the error text. That is deliberate. The model is your last line of recovery, not a helpless victim of a stack trace.

Roll your own vs LangGraph vs CrewAI

I get this question weekly. Here is the honest answer.

Roll your own if:

• Your agent has fewer than about 5 distinct roles
• You need tight control over cost, latency, and error surface
• You want to actually understand what your agent does when it misbehaves
• You are shipping to production and will maintain it for years

Use LangGraph if:

• You have genuine graph structure: parallel branches, cycles, checkpointing across a long-running workflow
• You want first-class human-in-the-loop primitives without building them
• Your team already knows the LangChain ecosystem

Use CrewAI if:

• You are prototyping a multi-agent concept quickly and the exact wiring does not matter yet
• Honestly, I have not shipped anything on CrewAI to production. It is fine for demos. I have always ended up rewriting.

The tradeoff nobody talks about: frameworks lock in a mental model. Once your team thinks in "Crews and Tasks" or "StateGraph nodes", it is very hard to think outside that model when you need to. Rolling your own means the mental model is: functions, state, loops. Universal.

For reference, the 200-line loop I showed at the top has been the base of every production agent I have shipped in the last two years, including the content pipeline that publishes across multiple sites unattended. I have never regretted starting small.

What I would do if starting a new agent today

1. Write the tools first. Type them, test them without an LLM. If your tools are broken, no model saves you.
2. Start with the 30-line loop. Get one end-to-end run working with a real task before you add anything.
3. Add the trace store. Postgres table, columns for run_id, step, role, content, tokens, cost, latency. Query with SQL.
4. Add budget caps. Steps, tokens, dollars. Non-negotiable.
5. Add loop detection and structured error returns. These two catch 80% of production weirdness.
6. Only then, if the task actually needs it, split into supervisor and workers.
7. Only then, if you have real graph structure, consider a framework.

The teams that ship fast are the ones that resist the urge to adopt heavy abstractions before they have felt the pain the abstraction solves. Every abstraction you adopt without earning it becomes a debugging tax later.

Closing

Production agents are not a framework problem. They are a systems problem: tool design, memory layering, orchestration discipline, and relentless failure handling. Get those right and a 200-line loop will outperform a 10,000-line framework wrapper you do not understand.

If you are building an agent that has to actually run in production and you want a second pair of eyes on the architecture, get in touch at lazar-milicevic.com/#contact. Or read more of what I have shipped and learned on the blog.