A RAG answer can faithfully quote a source and still be wrong because the source is outdated or does not apply to the user. Monitor document version and applicability alongside answer groundedness. A citation is useful evidence only when it points to the right material for the task.
This problem appears in return policies, pricing rules, product limits, internal procedures, and account-specific contracts.
Separate three kinds of correctness
A retrieval system must find relevant information. The selected information must be authoritative and applicable. The answer must accurately represent it. These checks answer different questions.
| Check | Failure example |
|---|---|
| Relevance | Search returns shipping policy for a cancellation question |
| Applicability and freshness | Search returns an old cancellation policy for the wrong region |
| Groundedness | Correct policy says 14 days; answer says 30 |
Microsoft’s RAG evaluator documentation distinguishes retrieval and generation evaluation. The freshness contract here adds the application-specific question of which source version should govern the answer.
Keep a source identity that survives editing
Record a document ID, version or content fingerprint, effective date where relevant, retrieval time, and applicability fields. A URL alone may point to a page that changed after the conversation. Saving the exact approved excerpt or a retrievable version helps later investigation.
Example application metadata:
{
"document_id": "returns-policy",
"version": "2026-10-01",
"effective_from": "2026-10-01",
"region": "GB",
"product_type": "standard",
"retrieved_at": "2026-10-20T10:00:00Z"
}These are illustrative fields. Do not infer applicability solely from the most recent timestamp. A contract signed earlier may legitimately use an older policy, and a future-effective policy may not apply yet.
Trace the freshness path
Inspect the original source, ingestion job, index, retrieval filter, application cache, and model context. A new document in the source repository does not mean every serving index has ingested it. A correct index does not help if a task cache serves yesterday’s result.
Use a controlled update in staging. Change a distinctive fact in a test document, publish the expected version, and measure when the serving path begins returning it. Repeat with the relevant tenant and region filters. This tests propagation, not just the indexing job’s success status.
If the agent continues to use old content, identify the first stale boundary. Changing the prompt will not repair a cache key that ignores policy version.
Build two kinds of tests
First, test retrieval against expected source identities. Second, test answer claims against the retrieved and applicable evidence. Include an older policy that remains valid for one cohort so the detector does not equate “older” with “wrong.”
Useful cases include a policy effective tomorrow, a missing region, two conflicting authoritative documents, and a cache entry surviving a source update. Define whether the agent should ask a clarifying question or abstain when it cannot select the correct source.
Keep the source snapshot with the regression case. Replaying against today’s mutable index can hide the original failure or create a different one.
Monitor the right population
Report the fraction of source-dependent answers with an applicable version recorded, confirmed stale-source findings, and unknown applicability. Review failures by document family and application version. A global average can hide one policy that affects only a small but important group.
In Agnost, connect the retrieval record to the conversation evidence. Include approved source metadata in your instrumentation so reviewers can distinguish an outdated source from a misread one. The answer-to-tool comparison guide covers the second case.
The resulting finding should say which policy was used, which policy should have applied, and what evidence establishes that difference. “The agent hallucinated” is too broad to tell the indexing, application, or prompt owner what to fix.