Observe First, Govern Second: What Brex Got Right About Agent

The first agent permission policy I ever wrote was a page and a half of careful rules. Within a week of the agent running unattended, it had hit three endpoints I'd never thought to list, and I had no idea which of them were legitimate. That is the trap with writing policy before you've seen behavior: you're governing an imagined agent, not the one actually running.

So I paid attention when Brex described how it solved the same problem for its own agents. Their approach, as reported by VentureBeat, is to watch what agents actually do and derive the policy from that, not to write rules first. It matches how I now roll out agent permissions in production, and I think it's the right default.

The problem: agents need real credentials

Agents need real credentials to be useful: API keys, OAuth tokens, service accounts. According to Brex's own write-up on building CrabTrap, the same agents can hallucinate destructive actions or be prompt-injected while holding those credentials. You can't solve this by withholding access, because an agent without access does nothing.

Brex's security team initially rejected deploying an agent platform internally, as Pedro Franceschi described at VB Transform. Instead of policing the code, Brex moved the security boundary to the network layer and monitored what the container sends and receives. That's the same premise behind how I think about agent security: assume the agent could do anything, and put your control point where every action has to pass through it.

The result is CrabTrap, an open-source HTTP/HTTPS proxy that intercepts all agent network traffic, checks it against policy, and uses an LLM-as-a-judge to approve or deny requests. Brex has open-sourced it.

Why rules-first fails with agents

Traditional guardrails assume a known surface. A web app has routes. A service has a fixed set of downstream calls. You can enumerate them, write allow rules, and be mostly right.

An agent's surface is whatever its tools, prompts, and the current task happen to produce. In my experience, writing rules first goes wrong in three ways:

1. You miss legitimate traffic. The agent calls an endpoint you didn't know existed (a pagination helper, a metadata lookup, a redirect target), and the policy blocks it. You loosen the rule in a panic, usually too far.
2. You over-permit defensively. Because you can't predict what's needed, you write broad wildcards, and now the policy permits exactly the destructive calls you wanted to stop.
3. You can't tell signal from noise. A blank-page policy has no baseline, so every denial looks equally suspicious or equally ignorable.

Franceschi's insight, per VentureBeat, was to bootstrap policy from observed behavior and edit down, which he found far more effective than starting from a blank page. Starting from what's real and subtracting is easier than starting from nothing and guessing.

The observe-then-govern loop

CrabTrap's design makes this practical. Without a config file, it starts in passthrough mode: all requests are allowed and logged. That's a natural observe-first starting point, and it's the posture I want for any new agent in a controlled environment. The quickstart is the place to check the current setup details.

The sequence I follow, and which Brex's approach mirrors, looks like this:

1. Run in shadow or passthrough mode. Log everything, block nothing. Keep the agent in a low-blast-radius environment while you do this.
2. Collect a meaningful window of real traffic. Brex pointed its policy builder at real traffic captured from agents doing actual work.
3. Cluster and sample. Group requests by host, method, and path shape, and sample representative calls from each group.
4. Draft the policy from what you see. Brex's policy builder is itself an agentic loop: it runs agents in shadow mode, analyzes historic network traffic, samples representative calls, and drafts a natural-language policy matching what the agent actually does.
5. Edit down. Remove anything that looks like accident, scope creep, or a call you'd never want repeated.
6. Replay before you ship. Test the draft against history before it goes live.
7. Enforce, then keep watching. Deny logs become your feedback signal.

Brex reported that the drafted policies matched human judgment on held-out requests more closely than they had expected. I'd treat that as Brex's own account, but it matches my experience that observed traffic gives you a draft far closer to reality than a blank page does.

Deterministic rules first, judge for the long tail

The part of the architecture I most agree with is the layering. CrabTrap matches deterministic static rules first. Deny rules always take priority over allow rules, and URL-pattern matching handles the bulk of known traffic. The LLM judge only handles the long tail: unfamiliar endpoints or unusual request shapes. For a mature agent, Brex says that's a small fraction of requests.

That matters for three reasons.

Cost and latency. Calling a model on every request would be wasteful. Brex says it used small, fast models for the judge, so the added latency was modest even when the judge fired.

Predictability. High-volume, well-understood traffic gets handled by rules you can read and test. Brex says static rules handle most of a mature agent's traffic, with the LLM judge covering only the long tail. Treat that as their experience, not a benchmark, but it's consistent with what I see: most agent traffic is boring and repetitive.

Judgment where judgment is needed. The judge receives the full request context plus the natural-language policy assigned to that agent, and returns a structured JSON decision (ALLOW or DENY) with a reason. A reason string on every decision is what makes the denial log usable later.

A sketch of how I structure the decision contract in my own agent gateways:

{
  "decision": "DENY",
  "reason": "Payment creation is outside this agent's scope",
  "matched": "judge",
  "policy_version": "<version identifier>"
}

The matched field (static rule vs. judge) and the policy version are my additions, not something taken from Brex. I find them essential for debugging, because "why did this get blocked last Tuesday" needs an answer that survives policy edits.

Replay is what makes policy changes safe

The piece that turns this from a clever demo into something you'd run in production is the eval step. Brex built a system that tests policy changes before they go live: it compares historical audit entries against a draft policy and reports the exact changes. Replaying a large batch of requests is reportedly fast enough to run on every change.

This is the same discipline I apply to prompt and pipeline changes in my own autonomous systems. In my content pipeline, a change to targeting or generation logic gets checked against past runs before it touches anything live. Policy deserves the same treatment. A policy edit is a code change with security consequences, and "I think this looks right" is not a test.

A replay report I'd want to see before approving any change:

"Newly allowed" is the row people skip, and it's the dangerous one.

Things I'd watch out for

CrabTrap is honest about its limits, and you should be too.

• The proxy sees everything in cleartext. It terminates TLS with a per-host certificate from a custom CA, and it does not redact sensitive data. Authorization and Cookie headers are visible to the proxy. The trust boundary is the proxy itself, so harden and isolate it like the credential-bearing system it effectively is.
• The judge is itself an attack surface. Brex structures each request as a JSON object before sending it to the model, so user-controlled content is escaped rather than interpolated as raw text. I'd do the same, and I'd still keep the judge's scope narrow: it should be deciding allow or deny, never taking actions.
• Observed behavior can include bad behavior. If an agent is already doing something it shouldn't during your observation window, a policy built from that traffic will bless it. This is why the "edit down" step is a human review, not an auto-accept.
• Shadow mode needs a safe environment. Passthrough means real requests succeed. Observe in a sandbox or with scoped credentials, not against production systems with destructive permissions.
• Don't treat the judge's output as proven accuracy. Measure your own false-allow and false-deny rates with replay rather than relying on anyone's reported numbers.

Observability pays for itself twice

One detail I liked: Brex stores full audit trails in PostgreSQL, queryable through an admin API and dashboard. And the denial logs and traffic analysis were used for more than security. They helped tighten agents, remove tools, and cut request categories that wasted time and tokens.

That's the hidden benefit of putting a governance layer on the network. You get a complete behavioral record of the agent, and that record shows you what the agent wastes effort on as well as what it does wrong. I've seen the same effect in my own multi-agent systems: once you can see every outbound call, pruning tools and trimming wasteful steps becomes straightforward, and the savings show up in token spend.

What I'd do

If I were rolling out a new agent with real credentials next week:

1. Put a proxy in front of it from day one, in log-everything mode, in a sandbox with scoped credentials.
2. Run it for a few days of real work before writing any policy.
3. Generate the draft policy from the traffic, then review it by hand and delete everything you wouldn't defend in an incident review.
4. Put deterministic rules first and the LLM judge second, with deny rules winning over allow rules.
5. Replay history against every policy change and read the "newly allowed" row first.
6. Keep the audit trail queryable, and review denials weekly. They tell you where the policy is wrong as often as where the agent is.
7. Treat the proxy as the most sensitive component in the stack, because it sees every credential in cleartext.

The general lesson goes beyond any one tool: agent policy is empirical. You learn the real surface by watching it, and you only get to write good rules after that. If you're facing this problem and want to compare notes, reach out at lazar-milicevic.com/#contact, or browse the rest of the blog for more on running agents in production.