Catch AI Agent Wrong Answers That Traces Miss

Traces tell you whether each step finished. This guide is about the other question: whether the answer was right. It gives you ten detectors, the metrics and alert rules built on them, and a four-week rollout plan. Each detector links to a deeper article if you need the full method.

Why the trace stays green

An OpenTelemetry span’s status says whether an operation completed or failed. That is the right question for a timeout or an exception. It is the wrong question for these cases, which all end with status OK:

The refund tool queued a request, and the agent told the customer the refund was complete.
Retrieval returned last quarter’s policy, and the agent quoted it accurately.
The search tool returned an empty list with HTTP 200, and the agent filled the gap with a plausible answer.
The agent returned valid JSON for the wrong order.

Each one passes four layers a trace can see (transport, tool execution, schema, latency) and fails the one it can’t: whether the claim matches reality. Why HTTP 200 does not mean your agent succeeded walks through those layers in detail.

The detector cookbook

A detector compares two pieces of evidence and returns pass, fail, or unknown. Start with the failure modes your users have already reported, not with all of them.

Detector (links to the full method)	Compares	Evidence you must capture	Detection method	Starting threshold
Claim contradicts tool result	Final answer vs. normalized tool output	Tool name, result state, final answer	Deterministic when the result is structured; judge otherwise	Any mismatch on a state field (`pending` vs. “done”)
Unsupported completion	“Done” claim vs. receipt in a terminal state	Action, entity ID, receipt state	Deterministic	Any completion claim without a matching terminal receipt
Ungrounded answer	Answer facts vs. retrieved passages	Retrieved chunks with document IDs	Judge, per claim	Any core claim with no supporting passage
Stale source	Retrieved version vs. current applicable version	Document ID, version, effective date	Deterministic	Any answer citing a superseded version
Wrong entity or tenant	Entity in the answer or tool args vs. the user’s scope	User and tenant IDs, tool arguments	Deterministic	Any cross-tenant entity: page immediately
Rule breach	Conversation vs. a written rule	Full conversation with tool calls	Judge, per conversation	Rate above the trailing 7-day baseline
User correction	Agent answer vs. the user’s next turn	Turn order within a conversation	Judge or classifier on the next user turn	Correction phrases (“that’s wrong”, “no, I meant”) within two turns
Re-asked intent	First request vs. later requests for the same goal	Conversation ID, intent per turn	Intent classification	Same intent asked again within three turns after an answer
Downstream reversal	Agent outcome vs. what happened next	Outcome events: ticket reopened, human override, refund reversed	Deterministic join on conversation ID	Reversal within 7 days of a “resolved” answer
Redundant successful reads	Tool calls within one conversation	Tool name, normalized arguments	Deterministic	More than two equivalent successful reads

The thresholds are starting points. Tune each one against reviewed production conversations, and store the detector version with every result so a threshold change never looks like a behavior change.

Pick the method by the evidence you have

Method	Use it when	Strength	Weakness
Deterministic check	The evidence is structured: a state, an ID, a version, a number	Exact, cheap, runs on every conversation	Only works on fields you captured; misses claims in free text
LLM judge	The claim is in prose, or the rule needs judgment	Reads whole conversations; one rule covers many phrasings	Costs money per run, drifts with its prompt, and must be calibrated against human labels
User and outcome signals	You can observe what the user or system did next	Catches failures no rule anticipated	Arrives late, and many users leave without correcting anything

Most teams need all three. Deterministic checks guard the actions that move money or data. Judges cover policy and grounding. User signals tell you which failure modes you haven’t written a detector for yet.

Capture the evidence first

A detector can only compare what you recorded. Add application-owned attributes to the spans or events your agent already emits:

app.agent.release          = "2026.11.02.1"
app.agent.intent           = "refund_order"
app.tool.result_state      = "pending"
app.tool.entity_id         = "order_42"
app.retrieval.doc_versions = ["refund-policy@v7"]
app.outcome.receipt_state  = "pending"
app.evidence_complete      = true

These names are examples, not OpenTelemetry semantic conventions. Use a prefix your organization owns. Hash identifiers that are sensitive but still need equality checks, and never attach secrets.

Recipes: from finding to metric

Every detector run should write one record with the same shape, whatever produced it:

conversation_id, detector_id, detector_version, verdict (pass | fail | unknown),
evidence_complete (bool), intent, release, evaluated_at

From those records, five metrics answer almost every question about silent failures.

Metric	Formula	What it tells you
Failure rate	`fail / (pass + fail)` per detector	How often this failure mode happens among conversations the detector could judge
Evidence coverage	`evidence_complete / eligible`	Whether the failure rate is trustworthy this week
Unknown rate	`unknown / eligible`	How much traffic the detector can’t judge; a rising value usually means broken instrumentation
Silent failure share	Conversations with at least one `fail` and zero error spans, divided by all conversations	How much of your quality problem your error dashboards never show
Detector precision	Confirmed findings / reviewed findings, from a weekly sample	Whether the detector deserves to page anyone

Keep unknown out of the failure-rate denominator. Folding it in hides failures when evidence collection breaks; counting it as failure creates false alarms.

Here is the daily failure rate and coverage per detector in plain SQL, assuming a detector_results table with the shape above:

SELECT
  date_trunc('day', evaluated_at)                                   AS day,
  detector_id,
  detector_version,
  count(*)                                                          AS eligible,
  count(*) FILTER (WHERE verdict = 'fail')                          AS failures,
  count(*) FILTER (WHERE verdict = 'fail')::float
    / nullif(count(*) FILTER (WHERE verdict IN ('pass', 'fail')), 0) AS failure_rate,
  count(*) FILTER (WHERE evidence_complete)::float / count(*)       AS evidence_coverage,
  count(*) FILTER (WHERE verdict = 'unknown')::float / count(*)     AS unknown_rate
FROM detector_results
GROUP BY 1, 2, 3
ORDER BY 1, 2;

Group by intent instead of day to find which user goals fail most, and by release to see whether a deploy moved the rate.

A dashboard that fits on one screen

Failure rate by detector over time, with release markers. This is the chart you look at first.
Evidence coverage and unknown rate, on the same time axis. Read it before trusting chart 1.
Failures by intent, as a ranked bar chart for the last 7 days. This is your fix queue.
Silent failure share, as a single number with its 30-day trend. This is the number for leadership.
Review queue: the latest findings with links to the conversation, so anyone can check the evidence in one click.

Leave out token counts, latency percentiles, and per-model cost on this screen. They belong on the operations dashboard, and mixing them in buries the quality signal.

Alerting playbook

Alert	Condition	Route to	Why
Severe single case	Any confirmed wrong-tenant answer, or a detector you marked severe	Page the on-call engineer	One occurrence is an incident
Rate rise	Failure rate over 24 hours is at least 1.5× the trailing 7-day rate, with at least 50 eligible conversations	Ticket to the agent’s owning team	Catches regressions after a deploy or a data change
Coverage drop	Evidence coverage falls more than 10 points in 24 hours	Whoever owns instrumentation; suppress rate alerts for that detector	A rate computed on missing evidence is noise
New failure pattern	A new intent or cluster of corrections appears in volume	Weekly review	Finds the failure modes you have no detector for

Two rules keep this from becoming noise. Never alert below a minimum eligible volume, because two findings in ten conversations look like a 20% failure rate. And put the detector version in every alert, so a threshold change is never mistaken for an incident.

A four-week rollout

Week	Do this	Done when
1. Capture	Add the evidence attributes for your top three failure modes. Run the detectors in shadow mode, with no alerts.	Evidence coverage is above 90% for each detector
2. Calibrate	Label 100 to 200 findings plus healthy controls. Measure precision and misses per detector.	Each detector you keep has known precision
3. Alert	Turn on rate and coverage alerts for detectors with at least 80% precision. Keep the rest on a review-only dashboard.	The first alert reaches the right owner with a link to evidence
4. Fix	Turn every confirmed pattern into a regression test, ship a fix, and compare the rate on the same detector version before and after.	One failure mode shows a lower rate after a release

Sample with care while you do this. A random 1% trace sample can hide a failure that affects one intent; keep targeted retention for flagged conversations alongside the representative sample.

From detection to a fix

A detector that fires every day without anyone changing code is a cost, not a safeguard. Give every confirmed pattern an owner, a regression case built from the real conversation, and a release whose effect you measure with the same detector version.

In Agnost AI, that loop looks like this:

Write the detector as a rule. Violations are rules in plain language, such as “Describe a refund as complete only after a completed refund receipt,” checked against each analyzed conversation with the supporting messages linked as evidence.
Scope the population with intents. Intents label what each conversation was trying to do, so you can see which user goals fail most.
Alert on it. Choose Alert on a violation or intent to fire on its count over a lookback window, or write a custom alert condition when you need something else, such as a rate.
Get a pull request. With Auto Improve connected to your repository, a fired alert starts a run that reads the evidence and your code and opens a pull request with the root cause, the change, and the expected impact. About an hour after you merge it, Agnost AI re-checks the alert condition and marks the fix improved, unchanged, or regressed.

Agnost AI judges what it receives: conversations, tool calls, and events. It cannot see a payment ledger or a CRM you don’t send it, so send the authoritative outcome as an event when a detector depends on it.

Other tools that cover part of this

Langfuse runs LLM-as-a-judge evaluators on individual observations in production and can alert on score rates, such as the share of true hallucination scores. Its trace-level evaluators are deprecated and stop producing results on Langfuse Cloud after November 16, 2026, so judge whole conversations elsewhere or score them through its API.
Arize AX runs online evals at span, trace, or session granularity with sampling, and its Signal agent groups recurring failures in production traces into ranked issues.

For a side-by-side on detectors and remediation, see Agnost AI vs Langfuse vs Arize for catching agent quality failures.

Frequently asked questions

Why don't wrong answers show up as errors in traces?

A span's status records whether an operation completed, not whether the answer was right. A tool can return HTTP 200 with an empty result, a model can answer fluently from a stale document, and every span still ends with status OK. Catching these needs a check that compares the answer with something else: the tool output, the retrieved source, the system of record, or the user's next message.

Should I use an LLM judge or deterministic checks?

Use deterministic checks wherever the evidence is structured, such as a receipt state, an entity ID, or a policy value, because they are cheap and exact. Use an LLM judge for claims in free text and for rules that need judgment, and calibrate it against human labels before alerting on it. User signals such as corrections and re-asks catch what both miss, but they arrive late.

What metric should I alert on?

Alert on a rate, not a raw count: failures divided by the eligible conversations for that detector, with unknown cases kept out of the denominator. Graph evidence coverage beside it. If coverage drops, a falling failure rate may only mean the detector stopped seeing evidence, so route a coverage drop to whoever owns instrumentation instead of celebrating the improvement.

How many conversations do I need before a detector's rate is meaningful?

Set a minimum eligible volume per alert window, such as 50 eligible conversations, and do not fire below it. Under that, one or two findings swing the rate enough to cause false alarms. Severe single cases, such as an answer about another customer's account, should page on any confirmed occurrence regardless of volume.

Can Agnost AI catch wrong answers that produce no error?

Yes, for failures visible in the conversation and tool evidence it receives. You write rules in plain language, Agnost AI checks each analyzed conversation against them and links the supporting messages, and alerts fire on violation or intent counts. When an alert fires, Auto Improve can open a pull request with a proposed fix. Agnost AI cannot see systems you do not send it, such as a payment ledger, so send the outcome as an event when a check depends on it.