TrueForge vs In-House Agent Harnesses: What Actually Saves Money

TrueFoundry dropped TrueForge last week, an Apache-licensed agent harness that claims 30% to 75% cheaper task completion than Claude's managed agents. I read the announcement, then spent a weekend running it against the harnesses I've built for my own production workloads. The honest answer is more interesting than the headline: open source harnesses save real money in a narrow set of cases, and quietly cost you more in others.

I've been building agent scaffolds since before "harness" was the accepted word for them. My ContentStudio pipeline at BizFlowAI runs a multi-agent loop 24/7, and I've shipped enough custom orchestration on AWS Lambda + EventBridge to have opinions about what's actually load-bearing in these systems. Here's what I'd tell a CTO evaluating TrueForge against a build-your-own path.

What TrueForge actually is, in one paragraph

TrueForge is a control layer that sits between your application and the model API, handling the loop of tool calls, state, retries and cost accounting. It gives you deterministic tool routing, a pluggable planner, and hooks for policy enforcement. The cost savings come from three places: aggressive prompt caching, cheaper models for planning steps that don't need frontier reasoning, and short-circuiting speculative tool calls before they hit the wire. None of these tricks are new. What TrueForge does is package them behind a clean interface so you don't have to write them yourself. If you've ever built a while loop that calls a model, parses a tool call, executes it, and appends the result to context, that's the shape.

Where the 30% to 75% number comes from (and where it doesn't apply)

Managed agents from Anthropic, OpenAI and Google are convenient because they hide the loop. You get one API call, you get a final answer. The trade-off is that the provider makes every routing decision for you, and you pay frontier-model rates on every internal step, including the boring ones (parsing a JSON payload, deciding which of two tools to call, summarizing a long observation).

TrueForge's savings come from breaking that loop open:

Stack them and 30% to 75% is credible on long-running, tool-heavy tasks. On a single-turn question that needs one model call and no tools, you save nothing. The savings show up when the managed agent would otherwise spin for 40 turns.

The trap: teams evaluate agent harnesses on toy benchmarks that are already short and cheap, then wonder why the savings evaporated in production. Measure on your actual task distribution, not on the vendor's demo.

My homegrown harness, and why I built it

Before I compare, some honesty about what I've actually built. My ContentStudio harness is around 900 lines of TypeScript, plus a Postgres schema for run state. It does:

• A supervisor loop that dispatches to specialist agents (research, draft, edit, SEO, publish)
• pgvector + full-text search with reciprocal rank fusion for grounding
• Deterministic tool routing (no letting the model pick from 40 tools; each agent sees at most 6)
• Prompt caching on Claude, with cache keys tied to the site config
• A JSONL event log per run, so I can replay any failure locally
• Cost accounting per run, per agent, per tool call, written to Postgres

It runs unattended, publishes across multiple sites, and self-corrects when a step fails. Total cost per published article, including research: usually under $0.40 on Claude Sonnet, sometimes as low as $0.12 when the cache is hot.

I didn't build this because I love reinventing wheels. I built it because when I evaluated the harnesses that existed at the time, each one enforced abstractions that fought my domain. LangGraph wants you to think in state machines. CrewAI wants roles. AutoGen wants conversations. My workflow is a DAG with conditional branches and it just wanted to be a DAG.

TrueForge vs my harness on three real workloads

I ran three tasks through both. Numbers are averaged over 10 runs each on Claude Sonnet 4.5, with the same tool definitions.

Workload 1: Research and summarize (3 web searches, 1 draft, 1 edit)

• My harness: $0.11 per run, 42 seconds
• TrueForge default: $0.09 per run, 51 seconds
• Managed Claude agent: $0.19 per run, 38 seconds

TrueForge wins on cost, my harness wins on speed (fewer round trips because tool routing is hardcoded). Managed agent is the most expensive but the fastest per token because it doesn't parallelize the planner/executor split.

Workload 2: Long-running content pipeline (12+ tool calls, 3 sub-agents)

• My harness: $0.34 per run, 3m 12s
• TrueForge default: $0.28 per run, 3m 48s
• Managed Claude agent: $0.71 per run, 2m 55s

Here TrueForge's savings compound. On a 12-step task, the planner/executor split matters. My harness is close because I've done the same optimization by hand.

Workload 3: Interactive debugging assistant (variable tool count, human in loop)

• My harness: $0.22 per session, but I built session state
• TrueForge default: $0.31 per session, session state is basic
• Managed Claude agent: $0.24 per session, best UX

TrueForge loses this one. Its session handling is thinner than the managed agents, and I already had session state built for my use case. This is exactly the "adds complexity you don't need" trap: if your workload is interactive and stateful, the harness's optimizations for autonomous loops don't help you.

The five hidden costs of an open source harness

The headline compare (harness cost vs managed cost) is misleading because it ignores the cost of owning the harness. From my notes:

1. Upgrade churn. TrueForge is on version 0.4. Between 0.2 and 0.3 the tool schema format changed. If you build against a pre-1.0 harness, budget one engineer-week per quarter for upgrades. That's real money.

2. Observability you have to build anyway. TrueForge emits events but the dashboard is minimal. To get production-grade tracing you'll wire in Langfuse, Arize, or your own OpenTelemetry pipeline. Add two to four weeks of setup, then ongoing hosting cost.

3. Policy and safety layers. Managed agents give you content filtering, jailbreak resistance, and abuse detection for free. On TrueForge, you either accept the model's default guardrails or you build a policy layer. For B2B SaaS with SOC 2 in scope, this is not optional.

4. Debugging deep failures. When a managed agent misbehaves, you file a support ticket. When TrueForge misbehaves at turn 34 of a 40-turn loop, you read the trace yourself. Your senior engineer's hours are not free.

5. The "we can fork it" fantasy. Teams tell themselves they'll fork the harness if the maintainer disappears. In practice nobody has the appetite to own a 15k-line agent framework. If you actually would fork it, you should just build your own smaller thing.

Multiply these across a year and the "cheaper" harness can easily cost more than a managed agent unless your token volume is high.

When to use TrueForge, when to build your own, when to stick with managed

Here's the decision matrix I'd hand a head of engineering:

The last row is where TrueForge actually shines and where the announcement understates its value. If you have 5 product teams each building an agent, giving them a shared harness is a genuine platform play. That's a stronger argument than the raw cost number.

What I'd do if I were starting a new agent project this quarter

1. Start with a managed agent for two weeks. Get a working end-to-end path. Don't optimize.
2. Measure your task shape. Average turn count, tool call count, token count per turn, cost per completed task. This is the data you need to make any real decision.
3. If your token bill is under $2k/month, stay on managed. The engineering cost of switching exceeds the savings.
4. If your bill is $5k to $50k/month and your task shape is stable, try TrueForge. Run a two-week bake-off on real production traffic. Measure end-to-end cost, latency P95, and failure rate. Include on-call load in the calculation.
5. If your task shape is unusual, build a small custom harness. Under 1000 lines, no framework, direct API calls. This is what I did and I don't regret it.
6. Never let the harness dictate your data model. Your event log, your run state, your cost accounting should live in your database, not in the harness. Harnesses come and go. Your data outlives them.

The mistake I see teams make: they adopt a harness because it's popular, then spend six months contorting their workflow to fit its abstractions. The right frame is that a harness is a library, not an architecture. If it fits, great. If it doesn't, a few hundred lines of your own code will serve you longer than a fashionable framework.

A note on the open source AI infrastructure trend

TrueForge is part of a wave: open source agent harnesses, open source RAG stacks, open source eval frameworks. This is healthy. It means the underlying patterns are mature enough that people can package them. It also means we're at the peak of the "which framework should we adopt" question, and past the point where that question matters as much as we think it does.

The winning teams I know are not the ones who picked the right harness. They're the ones who picked any reasonable one, kept it thin, and put their effort into evals, prompt discipline, and understanding their users. The harness is table stakes. The work is elsewhere.

---

If you're weighing an agent harness decision and want a second set of eyes from someone who has shipped these systems in production, or if you're mid-build and the numbers aren't adding up, I'm happy to talk. Reach me at lazar-milicevic.com/#contact or read more on the blog. I keep writing these because the field moves fast enough that last quarter's answer is often wrong by now.