Millisecond Agents, Second-Scale APIs: Closing the Gap

At VB Transform 2026, infrastructure leaders from LinkedIn, Walmart and Zendesk landed on the same uncomfortable truth: the models aren't the bottleneck anymore. The legacy systems the agents have to call are. I've been shipping agentic workflows into environments with 15-year-old ERPs and SOAP endpoints for a while now, and the gap between an agent that thinks in 200ms and a downstream API that answers in 2.4 seconds is where most projects quietly die. This is the piece nobody scopes properly, and it's the piece I spend most of my architecture time on.

Why the latency mismatch actually matters

An LLM call with tool use runs a planning loop. Every tool the agent invokes is a synchronous branch in that loop, and the model is billed and timed while it waits. A single 2-second legacy call inside a 6-step plan turns a 3-second interaction into a 15-second one, and now you have three compounding problems: user perception, token cost (context is still hot in memory on some providers), and cascade failures when the agent times out and retries.

The Transform panel framed it correctly. Agents expect the world to respond at model speed. Your world doesn't. The Oracle EBS instance answering in 1.8s on a good day is the same one the agent will hammer with parallel calls the moment you enable tool concurrency. I've watched a well-behaved agent take down an internal quoting API in under 90 seconds by doing exactly what it was told: fan out, gather, decide.

The fix is not "make the legacy system faster." You won't. The fix is architectural: put something between the agent and the legacy system that answers at agent speed, and let the legacy system update it at its own pace.

The pattern I actually ship: read-through cache with an event-driven refresh

Every serious agent I've built in the last two years uses some variant of this. It's boring and it works.

The shape:

1. Agent calls a facade API you control, not the legacy system directly.
2. The facade reads from a hot store (Redis, DynamoDB, or Postgres with a proper index) that returns in 5-30ms.
3. The hot store is populated and refreshed by event listeners and scheduled workers that talk to the legacy system out-of-band.
4. Writes from the agent go into an outbox table and are drained asynchronously into the legacy system, with the agent getting an optimistic acknowledgment and a job ID it can poll or subscribe to.

Here's what this looks like as a rough shape in AWS terms, which is where I do most of this work:

Agent -> API Gateway -> Lambda (facade)
                        |
                        +--> DynamoDB (hot read, single-digit ms)
                        +--> SQS (write outbox)
                                |
                                v
                         Lambda worker -> Legacy SOAP/REST
                                          (retries, backoff, DLQ)

EventBridge (scheduled + event-driven) -> Lambda refresher -> DynamoDB

The agent sees a clean tool that returns in 40ms. The legacy system sees a small, predictable, rate-limited traffic pattern from your workers. Nobody gets hurt.

Cache invalidation without lying to the agent

The hardest part of this pattern is not the cache. It's telling the agent the truth about how fresh the data is. Agents will confidently act on stale inventory counts, stale ticket statuses, stale prices. The Zendesk person on that panel talked about ticket state being the killer, and it is. A ticket the agent thinks is open was closed 400ms ago by a human, and now the agent is drafting a follow-up email that will embarrass everyone.

Three things I always do:

Return freshness metadata in every tool response. Not as a nice-to-have, as a required field.

{
  "ticket_id": "TCK-88214",
  "status": "open",
  "assignee": "maria.k",
  "_meta": {
    "source": "cache",
    "fetched_at": "2026-07-18T09:14:22Z",
    "age_ms": 1840,
    "confidence": "high"
  }
}

Then in the system prompt, the agent has an explicit rule: for state-changing decisions on data older than N seconds, call the refresh_* tool first. This costs you one extra round-trip on critical paths and saves you from acting on lies.

Push invalidation, don't pull it. If the legacy system can emit any kind of event (a webhook, a database CDC stream via Debezium, a message on a bus), catch it and evict the affected cache key immediately. For systems that can't emit events, I run a lightweight polling worker on a tight loop for the small set of records that are "in flight" (recently touched by the agent), and a slower background refresh for everything else. Two-tier polling. Cheap and effective.

Use TTLs that match the semantics, not a global default. Product catalog: 24 hours. Inventory: 30 seconds. Ticket status: 5 seconds if actively being worked, 5 minutes otherwise. A single global TTL is how you end up either lying to the agent or DDoSing your legacy system.

Event-driven writes: the outbox pattern for agents

Reads are the easy half. Writes are where agents actually cause damage, because the agent expects a synchronous confirmation and the legacy system wants a two-phase commit ceremony with a batch job at midnight.

The pattern:

1. Agent calls create_purchase_order(...).
2. Facade writes to an outbox table with a generated job_id and status queued, then returns {job_id, status: "accepted"} in under 50ms.
3. A worker picks up the job, calls the legacy system with proper retries, exponential backoff and a dead-letter queue.
4. On success, the outbox row flips to completed with the legacy system's real ID. On terminal failure, it flips to failed with a reason.
5. The agent has a check_job(job_id) tool, or better, subscribes to a completion event via a webhook you emit.

The critical detail: the agent must be trained to reason about pending state. Its next step after create_purchase_order should not be "email the vendor confirming PO-12345 was created," because PO-12345 doesn't exist yet. It should be "await job completion, then act on the returned PO number." I encode this directly in the tool description:

create_purchase_order:
  Creates a purchase order asynchronously. Returns a job_id.
  The PO does not exist yet when this returns. You MUST call
  check_job(job_id) or wait for a job.completed event before
  referencing the PO in any downstream action.

Tool descriptions are prompts. Write them like prompts.

The concurrency trap nobody warns you about

Modern agent frameworks (Claude with parallel tool use, OpenAI's parallel function calling, LangGraph's fan-out nodes) will happily fire 8 tool calls at once. Your legacy system will not happily receive them. I've seen a Salesforce sandbox rate-limit itself into a 15-minute timeout because an agent decided to enrich 12 accounts in parallel.

What I do:

• Rate-limit at the facade, not in the agent. Token bucket per legacy backend, shared across all agent sessions. Redis + a Lua script does this in about 20 lines.
• Coalesce identical in-flight requests. If two agent sessions ask for the same customer record within a 200ms window, the second request should attach to the first request's promise, not fire a second call. This alone cut backend load 40% on one project I ran.
• Circuit-breaker the legacy calls. When the backend is degraded, fail fast at the facade with a clear "backend degraded, using cached data from X seconds ago" response, and let the agent decide whether to proceed or ask a human. Don't let a slow backend take the agent down with it.

A small comparison of what these changes did on a real integration I worked on (customer support agent talking to a legacy ticketing system):

Same models. Same agent logic. Different plumbing.

The two things you must instrument from day one

If you take one thing from this post, take this: agents are distributed systems, and you cannot debug distributed systems without proper tracing. Log statements will not save you.

Trace every tool call with OpenTelemetry. Every span should have: agent.session_id, agent.step_number, tool.name, tool.args_hash, cache.hit, backend.name, backend.latency_ms. When something goes wrong at 3am, you want to open one trace and see the whole plan, every tool call, every cache hit or miss, every backend response time. I use Langfuse or Arize for the LLM side and standard OTel for the infrastructure side, and I stitch them together with a shared trace ID passed as a header.

Record the freshness of every piece of data the agent acted on. If the agent made a decision based on inventory data that was 45 seconds old, that fact needs to be in the trace. When the business asks "why did the agent promise shipping we couldn't fulfill," you need to be able to answer "because at the moment of decision, our inventory cache said 12 units, refreshed 45s ago, and the actual number at that moment was 3, updated 12s ago by a warehouse worker." Without this, you're guessing.

What I'd do if I were starting this Monday

If you're standing up your first serious agent project against a stack that includes anything more than 3 years old, here's the sequence I'd follow:

1. Map the tools before you write the agent. List every legacy system the agent will touch. For each: current p50/p95 latency, rate limits, write semantics (sync/async), event emission capabilities. This is a two-day exercise that saves two months.
2. Build the facade layer first, no agent involved. Prove you can serve reads in <100ms and accept writes in <50ms for the top 10 operations. Test it with a load generator, not with an agent.
3. Add caching with proper invalidation and freshness metadata. Not before, not after. Get this right when it's just APIs.
4. Then bolt on the agent. Now the agent thinks it's talking to a modern, fast system, because it is.
5. Instrument end-to-end from day one. Don't add tracing after you have an incident. You will have the incident before you have the tracing, guaranteed.
6. Run a chaos day before you ship. Deliberately degrade the legacy backend and watch what the agent does. If it doesn't degrade gracefully, fix that before real users touch it.

The mistake I see most often is teams treating the agent as the whole product and the infrastructure as an implementation detail. It's the opposite. The agent is a thin, replaceable layer of reasoning over a lot of very careful plumbing. The plumbing is the product.

Closing

The Transform panel got it right, and I'm glad three companies at that scale said it out loud. The story of agents in production for the next two years is not going to be about smarter models. It's going to be about the boring, hard, systems engineering work of making agents live comfortably on top of infrastructure that was never designed for them. Caches, outboxes, event streams, circuit breakers, tracing. This is the actual job.

If you're working through this on your own stack and want a second pair of eyes, or you'd rather have someone who's shipped this pattern a few times own the architecture, come find me at lazar-milicevic.com/#contact. More field notes on agents in production are on the blog.