The AI Agent Development Tools I Actually Use

Every agent I ship has to survive the same test: nobody is watching it at 3 a.m. That requirement strips most tooling decisions down to one question. When this thing fails quietly, will I find out, and can I reproduce it? Here is the stack I reach for, and where I have learned to be skeptical of my own picks.

The framework layer: LangGraph when state matters, the Claude Agent SDK when the task is open-ended

For agents that must survive restarts, pause for approval and resume, I use LangGraph. LangGraph 1.0 ships durable state, built-in persistence and human-in-the-loop pause and approval patterns, and those are the features that matter when a workflow runs unattended. A multi-agent content pipeline is a good example: research, draft, optimize, publish, each step with its own failure modes. I want every transition checkpointed, so a crash at step four resumes at step four, not at step one.

The second reason is stability. LangChain says the 1.0 releases of LangChain and LangGraph commit to no breaking changes until 2.0, and LangGraph 1.0 is an LTS release that stays active until 2.0 ships. After that it enters maintenance mode for at least one year (see the LangChain release policy and the 1.0 announcement). For code I expect to run for years without me, that policy matters more than any feature. The one migration note I would flag: langgraph.prebuilt is deprecated in favor of langchain.agents, and LangChain 1.0 introduces create_agent with create_react_agent deprecated. It also requires Python 3.10+.

When the task is open-ended and tool-heavy (read a repo, edit files, run commands, iterate), I do not hand-build a graph. I use the Claude Agent SDK. The Python package requires Python 3.10+ and bundles the Claude Code CLI, so there is no separate install to manage (details in the SDK repo). It was formerly called the Claude Code SDK, so older tutorials and imports may still use the old name. I deliberately do not pin my mental model to a version number here, because the SDK moves fast. I pin versions in lockfiles and read the changelog before every bump.

My rule of thumb:

That last row is where I see the most waste. If your "agent" is one prompt and one parse step, you do not have an agent. You have a function.

Evals: Braintrust for gating releases, with eyes open on the bill

I treat evals as a deployment gate, not a dashboard. Before a prompt or model change reaches the scheduled worker, it runs against a fixed dataset, and a score regression blocks the release. For that workflow I have preferred Braintrust, mainly because experiments, datasets and scorers are first-class and comparing two runs side by side is fast.

The cost model deserves a read before you commit. Braintrust has three plans: Starter at a $0 platform fee, Pro at $249 per month, and Enterprise with custom pricing. Starter includes 1 GB of processed data per month (then $4/GB) and 14-day retention. Pro includes 5 GB per month (then $3/GB) and 30-day retention. Scores are metered separately: Starter includes 10k per month, then $2.50 per 1k, while Pro includes 50k per month, then $1.50 per 1k (see the Braintrust plans and limits). A third-party summary notes there is no per-seat charge, so cost is driven by processed data and scores rather than headcount. That is friendly for a small team, and it also means a chatty LLM-as-judge scorer run over a big dataset is where your bill actually comes from. Check the live pricing page before budgeting, since plan details change.

LangSmith is the obvious alternative, and if your stack is already LangGraph it is the lowest-friction option because the tracing hooks are native. It has a free Developer plan, a Plus plan at $39 per seat per month, and a custom Enterprise tier, per a third-party summary that should be checked against the vendor's pricing page. One trade-off to plan around: evaluation runs generate billable traces, so frequent evals over large datasets add to the monthly bill. I will not quote trace overage rates here because the sources I found conflict, and you should read the official LangChain pricing page rather than trust any blog, including this one.

The part that matters more than the vendor: your eval dataset is the asset, not the platform. I keep datasets as versioned files in the repo and sync them to whatever tool I am using. If I switch tools, I lose a dashboard, not two months of curated failure cases.

Observability: traces first, because agents fail in the middle

An agent that returns a wrong answer is easy to spot. An agent that takes a wrong turn in step three and still produces a plausible-looking output is the one that costs you. So every run gets a full trace: each model call, each tool call, inputs, outputs, latency and cost.

For this I lean on Langfuse when I want control over where the data lives. Its core is MIT-licensed and self-hosting is free regardless of Cloud tier, which matters if prompts or tool outputs contain customer data that should not leave your infrastructure (see this Langfuse review). Langfuse Cloud has a free Hobby tier and paid tiers. Langfuse is also now part of ClickHouse, which for me is a mild positive signal about longevity, though acquisitions always warrant watching the roadmap.

The billing gotcha is real and easy to miss: Langfuse Cloud bills in units (traces plus observations plus scores), so a multi-step agent run consumes many units per request. A ten-step agent with a scorer attached is not one trace, it is a dozen units. If you estimate cost by counting requests, you will undershoot. Count steps. Tier details differ between sources, so I would verify them on langfuse.com/pricing rather than rely on anything I write.

What I put in every trace, whatever the tool:

• A run ID that also appears in my application logs, so I can jump between systems
• The prompt version and model identifier used for that step
• Token counts and a computed cost per step
• A terminal status that distinguishes "completed", "completed with fallback" and "gave up"

That last field has saved me more than any dashboard. "Completed with fallback" is the early warning that a primary path is degrading before it fails outright.

Retrieval: pgvector until it hurts, then reassess

For retrieval, my default is Postgres with pgvector, usually on Supabase. I have built local-LLM RAG pipelines with hybrid search, combining vector similarity and keyword search with reciprocal rank fusion (RRF), and doing that in one database is a genuine simplification. Your documents, metadata, access rules and embeddings live in the same place, so you filter with plain SQL and join against your real tables. There is no second system to sync and no second place for data to go stale.

A managed vector database makes sense when retrieval is the product and you need operational features at a scale where running your own index is a distraction. Pinecone is the one people usually compare. A third-party source puts Pinecone Standard at a $50 per month usage minimum with storage at about $0.33/GB per month, plus separately metered read and write units. I would not rely on that article's comparisons for a purchase decision. Vector database cost estimates across the sources I checked varied wildly, and the read and write unit prices were inconsistent, so model your own query volume and use the vendor's current numbers.

My honest take: most business agents I build have corpora in the thousands to low millions of chunks, and the bottleneck is chunking and retrieval quality, not the index. I would rather spend a week on better chunking, metadata filters and a reranking step than a week migrating vector stores. Pick pgvector, measure recall on a labeled query set, and only move when the measurements say so.

Deployment: serverless, boring scheduling, and a kill switch

Unattended agents mostly fall into two shapes: event-triggered (a ticket arrives, a webhook fires) and scheduled (nightly, hourly). Both map cleanly onto serverless. My integration between AWS and Zendesk, which delivered the first-ever SLA compliance for that team, was a serverless design with event-driven triggers and minimal cost at idle. On AWS, EventBridge handles scheduling and API Gateway handles inbound events, with Lambda doing the work.

The caveat is that Lambda's execution limits do not suit long agent loops. My pattern is to keep each invocation short, persist state between steps (this is where LangGraph's checkpointing earns its keep), and let the next trigger pick up where the last left off. Long-running agent jobs go in a container instead. Docker on a small instance is unglamorous and perfectly reliable.

Three things I put around every deployed agent:

1. A budget cap per run and per day. A runaway loop is a billing event, not just a bug.
2. A kill switch that is a single config flag, checked at the start of each step, so I can halt without redeploying.
3. An idempotency key on every side effect. Retries are inevitable, and a duplicate email or duplicate publish is the failure users actually notice.

What I'd do

If you are starting an unattended agent today, this is the order I would build in:

1. Write the eval dataset first, even ten cases, kept as files in the repo.
2. Choose the framework by the shape of the work: LangGraph for staged, resumable workflows, the Claude Agent SDK for open-ended tool use, a plain API call for anything simpler.
3. Add tracing before you add features. Self-host Langfuse if data residency matters; use LangSmith if you are all-in on LangGraph.
4. Start retrieval on pgvector with hybrid search, and measure before you migrate.
5. Wrap deployment in a budget cap, a kill switch and idempotency keys.
6. Re-check every price and version above against the vendor's live page before you commit. This space reprices and renames itself constantly.

The pattern across all of it: pick tools that make failure visible and reversible, and keep your valuable artifacts (datasets, prompts, traces' schema) portable so no vendor decision becomes permanent.

If you are weighing a stack for an agent that has to run without supervision, I am happy to compare notes. You can reach me at lazar-milicevic.com/#contact, or keep reading on the blog.