The Agentic Context Layer: Fixing Confidently Wrong Agents

Last quarter I sat with a head of data who pulled up a Slack thread from their sales ops team. Their internal agent had confidently told a VP that Q2 pipeline was up 34%. It was up 12%. The model was fine. The agent had pulled a definition of "pipeline" from a 2023 wiki page that included canceled opportunities, while the finance team had quietly redefined it eight months earlier in a different Notion doc the retrieval system had never indexed.

Nobody caught it for three days. When they traced it back, the failure was not hallucination in the classic sense. It was a context failure. The agent answered from what it was given, and what it was given was stale, partial, and internally inconsistent.

A recent VentureBeat piece put a number on this: 57% of enterprises have traced a confidently wrong agent answer back to missing or inconsistent business context, and 31% say it happens regularly. The industry has a name for the fix now, an "agentic context layer," but almost nobody has actually built one. Here is how I build them.

What an agentic context layer actually is

An agentic context layer is the system between your raw data sources and your agent that decides what business truth the agent is allowed to see, in what form, with what provenance, and under what freshness guarantees. It is not RAG. RAG is one component inside it. The layer sits above retrieval and below the agent's reasoning loop, and it does four jobs: it resolves semantic definitions (what does "revenue" mean today), it enforces retrieval guardrails (never answer from a doc older than X for financial queries), it stamps provenance on every chunk, and it exposes eval hooks so you can measure context quality, not just answer quality.

Most enterprise "AI platforms" I audit have none of this. They have a vector store, a system prompt, and a model call. That is a demo, not a production system. The gap between the two is where confidently wrong answers live.

Why plain RAG produces confidently wrong agents

Vanilla RAG optimizes for one thing: retrieve the top-k chunks most semantically similar to the query. That is not the same as retrieving the chunks that are currently true. Three failure modes I see constantly:

1. Definition drift. The same term ("ARR", "active user", "customer") has three definitions across three docs written in three quarters. Similarity search returns all three. The agent picks one, usually the wrong one, and never mentions the conflict.
2. Retrieval blindness. A doc exists but was never chunked, or it was chunked but lives in a source the pipeline forgot to include. The agent does not know it does not know. It answers from what it has.
3. Freshness collapse. Finance updated the metric on Monday. The embedding job runs Sunday. For six days the agent confidently serves last week's truth.

None of these are model problems. Swapping Claude for GPT does nothing. You have to fix the layer above the model.

Step 1: Build a semantic definition registry

The first thing I build on any serious agent project is a definition registry. It is a small structured store, usually Postgres with pgvector, that holds the canonical business definitions the agent is allowed to use. Every entry has an owner, an effective date, a version, and a machine-readable formula where possible.

Rough schema:

create table semantic_definitions (
  term text not null,
  definition text not null,
  formula text,
  owner_email text not null,
  effective_from date not null,
  effective_to date,
  version int not null,
  source_url text,
  embedding vector(1536),
  primary key (term, version)
);

The agent does not get to guess what "pipeline" means. Before it retrieves anything else, a preflight step extracts business terms from the user's query, looks them up in the registry as of today, and injects the canonical definitions into context with a hard instruction: use these definitions, and if a retrieved document conflicts, flag it.

The second-order effect is political, not technical. Once you have a registry, someone has to own each definition. That forces a conversation between finance, product, and data that most companies have been avoiding for years. The registry is worth building for that alone.

Step 2: Retrieval guardrails that know what they do not know

Similarity is not enough. Every retrieval call in my systems goes through a guardrail wrapper that enforces at least four rules:

• Freshness floor per query type. A financial query cannot be answered from a doc older than 90 days without an explicit warning. A policy query can go back further. This is a routing decision, not a model decision.
• Source coverage check. For each query intent, there is a required set of sources. If the top-k results do not include at least one chunk from each required source, the agent is instructed to say "I do not have coverage of source X, my answer may be incomplete" rather than answer.
• Conflict detection. If two retrieved chunks contain contradictory numbers or definitions for the same entity, the agent must surface the conflict, not silently pick one. I do this with a lightweight second LLM call that only compares chunks pairwise.
• Hybrid retrieval as default. Dense embeddings plus BM25, fused with Reciprocal Rank Fusion. Pure vector search misses exact-match terms (SKU codes, account IDs, metric names) constantly. RRF is a two-line change that eliminates a large class of "why did it not find the obvious doc" failures.

Here is the shape of the wrapper I reuse across projects:

def guarded_retrieve(query, intent):
    dense = vector_search(query, k=20)
    sparse = bm25_search(query, k=20)
    fused = rrf_fuse(dense, sparse, k=10)

    coverage = check_required_sources(fused, intent)
    freshness = check_freshness_floor(fused, intent)
    conflicts = detect_conflicts(fused)

    return {
        "chunks": fused,
        "warnings": {
            "missing_sources": coverage.missing,
            "stale_chunks": freshness.stale,
            "conflicts": conflicts,
        }
    }

The warnings are not thrown away. They go into the agent's prompt as structured context, and they go into the eval log as features. When an answer is wrong later, I can immediately see whether the guardrail fired and was ignored, or never fired at all.

Step 3: Provenance on every chunk, all the way to the answer

Every chunk in my retrieval index carries a provenance record: source system, source URL, author, last modified date, and a hash of the content. When the agent produces an answer, it must cite chunks by ID, and the response envelope carries the full provenance for each citation.

This is not for the user. This is for you, six weeks later, when someone forwards you a bad answer and asks what happened. Without provenance, root-causing a bad agent answer is archaeology. With it, you have a two-minute lookup: what did the agent see, from which doc, last updated when, owned by whom.

A concrete number from one of my projects: after adding provenance and a small "why this answer" panel that users could expand, self-reported trust in the agent went up meaningfully and, more importantly, the number of bad answers escalated to the AI team dropped by roughly two thirds. Most of the previously "wrong" answers were actually right but had looked wrong to the reader. Provenance made them defensible. The remaining third were genuinely wrong, and now we could actually fix them.

Step 4: Eval hooks that measure context, not just answers

Most eval frameworks I see measure the final answer against a golden set. That is necessary but not sufficient. For an agentic context layer, I add three eval dimensions above the answer:

The interesting one is context faithfulness. It catches a specific failure mode: the retrieval was good, the definitions were right, but the model reasoned past them and added something it invented. That is a prompt and model-selection problem, not a context problem, and separating it in the eval tells you which lever to pull.

I run these evals on every prompt change, every index rebuild, and every model swap. They run in CI. If context faithfulness drops below 0.95 on the eval set, the deploy fails. This is table stakes for production, and almost nobody does it.

Step 5: A freshness pipeline that treats the index as a live system

The index is not a batch artifact. It is a live production database that happens to be reindexed. I run change-data-capture from the source systems (Notion, Confluence, Salesforce, the data warehouse) into a queue, and reindex only the chunks that changed. For high-signal sources (the definition registry, finance dashboards, exec-approved policies), the pipeline runs within minutes of a change. For low-signal sources (old wiki pages), nightly is fine.

Two details that matter:

• Soft-delete before hard-delete. When a source doc is deleted or superseded, I mark its chunks as deprecated for 24 hours before removing them, and log any retrieval that hits a deprecated chunk. This catches "the agent used to know this and now it does not" regressions before users do.
• Reindex diffs go to a channel. Every reindex posts a summary to Slack: docs added, docs deprecated, definitions changed. It sounds trivial. It has caught silent breaking changes maybe a dozen times across my projects.

What I would do if I were starting Monday

If a CTO asked me where to start, in order:

1. Stop shipping new agent features for two weeks. Ship a definition registry with the top 30 business terms your agents use. Get owners on each.
2. Add provenance to every retrieved chunk and every answer. This alone will change your team's relationship with the agent from "hope" to "audit."
3. Wrap retrieval in a guardrail that enforces freshness floors and source coverage per query intent. Log the warnings. Do not throw them away.
4. Build a gold set of 100 real queries with human-labeled correct sources and correct answers. Run context-faithfulness and retrieval-recall evals in CI.
5. Only then, add more agent capability. New tools, new sources, longer horizons. The context layer is what lets any of that be safe.

The order matters. Teams that skip to step 5 are the teams whose head of data pulls up a Slack thread six months later.

Where this leaves us

Agents are not confidently wrong because models are bad. They are confidently wrong because we hand them incomplete, stale, or contradictory context and then act surprised when they answer from it. The fix is unglamorous engineering: definitions, guardrails, provenance, evals, freshness. It is exactly the kind of work that separates a demo from a system a VP will trust with a real decision.

If you are building or buying an enterprise agent right now and you can feel the context problem but do not yet have a plan for it, that is worth a conversation. You can reach me at lazar-milicevic.com/#contact, or read more on how I think about production AI systems on the blog. The teams that solve this in 2026 are the ones whose agents get promoted from novelty to infrastructure.