Compare an agent’s factual claims with the exact tool evidence available when it answered. Check the subject, conditions, units, and applicable version, not only whether the response repeats a number found somewhere in the source.
If a policy lookup returns 14 days and the assistant says 30, the mismatch is easy to see. The harder cases involve different products, countries, customer tiers, or effective dates. A source can contain both numbers while only one applies.
Preserve the evidence the model actually saw
Keep the retrieved document identifier, version, relevant excerpt, retrieval arguments, and the context passed to the generation step. A raw search result is not necessarily the same content the model received after truncation or reranking.
When content cannot be retained, preserve approved structured facts and a resolvable source reference. Be explicit about the resulting limits: a reference to a document you can no longer retrieve is weak evidence for a later review.
Microsoft’s RAG evaluation guidance treats retrieval quality and answer groundedness as distinct evaluation concerns. That separation is useful here: a wrong source and a wrong interpretation call for different fixes.
Compare atomic claims
Break the answer into factual statements rather than scoring the paragraph as one unit. For a returns answer, the claims may be a deadline, the start date for that deadline, eligible item condition, and who pays shipping.
| Claim | Applicable evidence | Assessment |
|---|---|---|
| Return within 14 days of delivery | Policy says 14 days from delivery | Supported |
| Return within 14 days of purchase | Policy says 14 days from delivery | Wrong start condition |
| Every item is eligible | Policy excludes personalized items | Unsupported generalization |
| Refund already issued | Policy describes eligibility only | Action claim needs separate execution evidence |
This makes the finding actionable. “Groundedness score: 0.6” alone does not tell an engineer whether to fix retrieval, context construction, or a completion statement.
Choose the simplest check that fits the claim
Use deterministic comparisons for normalized dates, amounts, IDs, and enumerated states. Use a model-based reviewer for semantic claims that cannot be reduced reliably to fields. In either case, include the source and define an insufficient-evidence outcome.
A proposed reviewer instruction is:
Assess each factual claim using only the supplied source and task context.
Return supported, contradicted, or insufficient evidence. Identify the
source passage and explain any mismatch in subject, condition, or value.
Do not use general knowledge to fill a missing policy detail.Treat the returned assessment as another model output to calibrate. It can miss negations, over-trust plausible prose, or confuse a quotation with the assistant’s own claim. Test it on labeled examples before using it to drive alerts.
Diagnose the failing stage
If the correct policy never appeared in the retrieved set, inspect indexing, filters, and query construction. If it appeared but was removed before generation, inspect context selection and token limits. If the model saw it and contradicted it, inspect the instructions and response construction.
If the policy was correct yesterday but invalid today, investigate source freshness. A grounded answer can still be outdated. If the answer concerns an external action, use action verification, because reading a policy cannot prove a refund executed.
Build a regression set that resists shortcuts
Vary the policy window, country, excluded item types, and date basis. Include a source that legitimately says 30 days so a detector cannot succeed by flagging that number unconditionally. Include cases where the correct response is to ask for missing context.
Save the evaluator version with each result. Compare both the number of contradictions and the fraction of claims with sufficient evidence. A system that avoids all factual answers may reduce contradiction counts while becoming less useful.
In Agnost, use the conversation and tool-call records to reconstruct that evidence. Review a finding alongside its source before treating it as a confirmed product issue, then preserve the case in your regression dataset.