The AI Agents Course I'd Build From Production Experience

Most agent tutorials end at the moment the demo works. The real course starts there, when the agent runs unattended at 3 a.m., a tool returns something strange, and nobody is watching the logs. I've shipped autonomous systems that run on schedules with no human in the loop, and the gap between "it worked in the notebook" and "it runs every day for a business" is where almost all the learning lives. If I were designing a curriculum for engineers who want to build production-grade agents, this is the order I'd teach it in, and why.

Module 1: Start with a workflow, then earn the agent

The first lesson is the least exciting one: most of what people call an agent should be a deterministic pipeline with one or two LLM calls inside it. I'd open the course by making students build the boring version first.

My own ContentStudio system is the template for this. The loop (measure, learn, target, generate, publish) is a fixed sequence of stages. Within it, individual steps like research and drafting have room to make decisions, but the overall control flow is code I wrote and can reason about. I did not hand a model a goal and hope.

The decision rule I'd teach:

Anthropic's engineering team makes the same argument in their guide on building effective agents, and it's worth assigning as reading: anthropic.com The point I'd hammer is that every step toward autonomy trades predictability for flexibility. Students should be able to say what they bought with that trade before they make it.

Exercise: take a task, build it as a straight pipeline, measure where it fails, and only then decide which single step deserves to become an agent loop.

Module 2: Tool design is the actual work

If I had to put one idea on the whiteboard for the whole course, it would be this: your tools are your agent's real interface, and most of your engineering time belongs there.

Anthropic reported that while building their SWE-bench coding agent, they spent more time optimizing the tools than the overall prompt. One concrete fix they describe: requiring absolute filepaths, because the model started making mistakes with relative paths once the agent had moved out of the root directory. I've seen exactly this class of bug. It isn't a reasoning failure, it's an interface that invites ambiguity.

What I'd teach, in practice:

1. Make the wrong call hard to express. If a parameter has three valid formats, the model will eventually use a fourth. Pick one format and enforce it with a schema.
2. Return errors the model can act on. "Failed" teaches nothing. "Date must be ISO 8601, received '15/10'" lets the agent self-correct on the next turn.
3. Keep tool outputs small and relevant. Dumping a 40-row API response into context burns tokens and buries the signal. Return what the next decision needs.
4. Name things like you're writing for a new hire. Tool names and descriptions are prompt text. Vague names produce vague usage.
5. Make destructive tools separate and explicit. Never overload one tool with read and delete modes.

A tiny example of a tool contract that survives contact with a model:

def update_ticket(ticket_id: str, status: Literal["open", "pending", "solved"]) -> dict:
    """Set the status of an existing support ticket.
    ticket_id: the full ID string, e.g. 'TCK-10482'. Never a URL.
    Returns {'ok': True, 'ticket_id': ..., 'status': ...} or
    {'ok': False, 'error': '<what to fix>'}.
    """

The enum removes a whole category of invented values. The docstring specifies the ID format. The return shape is predictable. That's twenty seconds of design that saves hours of debugging.

I'd also cover the Model Context Protocol here, since it's becoming the standard way to expose tools. The 2026-07-28 spec release made the protocol core stateless, replacing the earlier stateful, bidirectional design, and deprecated the Sampling client feature (https://blog.modelcontextprotocol.io/posts/2026-07-28/). Students should learn to treat the protocol surface as something that moves, and keep their own tool logic behind a thin adapter so a spec change doesn't ripple through everything.

Module 3: Orchestration, and when multi-agent is a mistake

I'd teach multi-agent last in the orchestration block, because it's the most overused pattern in the field.

The cost is real and measurable. Anthropic reports that in their data, agents typically use about 4x more tokens than chat interactions, and multi-agent systems use about 15x more (https://www.anthropic.com/engineering/multi-agent-research-system). Fifteen times is not a rounding error. It changes the unit economics of a feature. If a single agent with good tools solves the problem, adding agents is just adding a bill.

So when does it earn its cost? In my experience, when subtasks are genuinely independent and breadth matters more than coherence: researching several separate questions in parallel, say. It fails when subtasks need to share a consistent picture of the world, because separate agents make separate assumptions and the outputs don't compose cleanly.

Two orchestration shapes I'd have students implement and compare:

• Orchestrator-workers: one lead decomposes the task, delegates narrow jobs, and synthesizes. Control stays centralized, which makes it debuggable.
• Handoffs: control passes to a specialist that takes over the conversation. The OpenAI Agents SDK documents this as a decentralized pattern, where an agent declares its targets through its handoffs list (https://openai.github.io/openai-agents-python/agents/). It fits triage-style flows well, and it's harder to trace than a central coordinator.

The framework itself matters less than people think. I'd tell students to learn the patterns on one SDK, then port one project to a second, so they see which ideas are portable and which are framework furniture.

Module 4: Memory and context as an engineering budget

Memory is where agents quietly rot. The naive approach, appending everything to the conversation, works for ten turns and fails at a hundred.

I'd teach context as a budget with three tiers:

• Working context: what the model sees this turn. Keep it tight and rebuilt deliberately, not accumulated by accident.
• Session state: structured facts the run needs (current plan, completed steps, open questions), stored outside the prompt and re-injected as a compact summary.
• Long-term memory: retrieval over past runs or documents. This is where RAG, pgvector, and hybrid search with reciprocal rank fusion come in. Dense and keyword retrieval fail in different ways, and fusing them is cheaper than tuning either one to perfection.

The habit I'd drill is externalizing the plan. A long-running agent should write its plan and progress to durable state and be able to resume from it after a crash, a timeout, or a deploy. My scheduled workers are built so that any run can be killed at any point and restart without redoing or duplicating work. That idempotency requirement shapes the whole design, and it's something tutorials almost never mention.

Exercise: kill a running agent halfway through a task. If it can't pick up where it left off, the memory design isn't done.

Module 5: Evaluation, observability, and cost control

This is the module that separates a hobby from a system, and I'd give it the most course time.

Observability first. You can't fix what you can't see. The OpenAI Agents SDK enables tracing by default, recording LLM generations, tool calls, handoffs, guardrails, and custom events (https://openai.github.io/openai-agents-python/tracing/). That's a good default, with one gotcha worth a whole lesson on its own: trace_include_sensitive_data defaults to True, so traces capture LLM inputs and outputs and function-call inputs and outputs unless you turn it off, in RunConfig or through the OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA environment variable (https://github.com/openai/openai-agents-python/blob/main/docs/tracing.md). If your agent touches customer tickets or personal data, that default can quietly ship it to a tracing backend. I'd have students audit what their traces contain before anything goes to production.

Evaluation second. My approach is unglamorous:

1. Collect real failures from logs. Every bad run becomes a test case.
2. Build a small fixed set of tasks with checkable outcomes, preferring assertions on side effects (the ticket was updated, the file exists) over grading prose.
3. Use an LLM judge only where there's no checkable outcome, and spot-check the judge against your own labels.
4. Run the set on every prompt, tool, or model change. Treat a regression like a failing unit test.

I'd deliberately avoid quoting benchmark scores in the course. Public leaderboards tell you about a model on someone else's tasks. Your eval set tells you about yours.

Cost control third. Given the token multipliers above, I'd teach hard budgets: a maximum number of turns, a maximum spend per run, and a circuit breaker that stops the agent and alerts a human rather than looping. An unattended agent without a budget is an open invoice.

Module 6: Deployment, safety, and the boring infrastructure

The last module is where my cloud background does the most work. Agents are just workloads, and the best deployment shape is usually the one that scales to zero when idle: scheduled triggers, event-driven functions, queues in between. My serverless AWS and Zendesk integration, which delivered first-ever SLA compliance, was reliable because it was ordinary event-driven plumbing with an LLM step in it, not because it was clever.

Topics I'd cover:

• Sandboxing. Anything that executes code or touches a filesystem runs isolated, with the narrowest credentials that work. Give each tool its own permission scope.
• Human-in-the-loop gates on irreversible actions: sending, deleting, paying, publishing. Autonomy should be earned per action type, not granted globally.
• Idempotency and retries. Assume every step runs twice someday.
• Regulation. For anyone shipping into the EU, the Commission's AI Act timeline says Article 50 transparency rules start to apply on 2 August 2026, with enforcement on GPAI models, prohibitions, transparency rules, and AI literacy also beginning then, and the main milestones fully rolled out by 2 August 2028 (https://ai-act-service-desk.ec.europa.eu/en/ai-act/timeline/timeline-implementation-eu-ai-act). I'd tell students to read that page directly rather than trust a summary, because the dates have been amended and they should work from the official source.

What I'd do

If you're an engineer building your first production agent, here's the short version:

• Build the pipeline before the agent. Add autonomy one step at a time, and only where you can state what it buys.
• Spend your effort on tools, not prompts. Tight schemas, actionable errors, small outputs.
• Distrust multi-agent by default. Pay the token multiplier only when subtasks are truly independent.
• Externalize state so any run can die and resume.
• Build the eval set from real failures on day one, and audit your traces for sensitive data before launch.
• Put a budget and a kill switch on everything that runs unattended.

None of this is glamorous, and that's the point. The agents that survive in production are the ones built by people who obsessed over the dull parts.

If you're working through a problem like this and want a second pair of eyes, you can reach me at lazar-milicevic.com/#contact, or browse the rest of the blog for more write-ups from real builds.