Your eval suite is green. Your demo works. The agent passed the 40 golden tasks you lovingly hand-crafted after midnight with a cold coffee and too much optimism.
Then production happens.
A production agent failure is a real user session where the agent runs, often with no errors, but still fails the job the user expected. The model returns text. The tools fire. But the user leaves confused, repeats themselves, fixes the work manually, or stops trusting the product.
That is the part evals miss.
Evals are useful. You should run them. But most evals are tests of known examples in clean rooms. Production is weird prompts, half-specified intent, stale memory, tool permissions, vague emotional temperature, and “nevermind I’ll do it myself.”
^ your eval dashboard while a power user rage-edits the agent output in another tab
Why do evals miss production agent failures?
Because evals usually measure answer correctness. Production failures are often experience failures.
The agent may answer the test prompt perfectly and still fail in the wild because the wild has missing context, user impatience, old project state, permission boundaries, and follow-up behavior. The failure is not always “bad answer.” Sometimes it is “good answer to the wrong problem.”
Here is the shortest version:
| What evals are good at | What production exposes |
|---|---|
| Known inputs | New intent shapes |
| Single-turn quality | Multi-turn trust |
| Correctness checks | User acceptance |
| Prompt regression | Product drift |
| Model behavior | Whole-agent behavior |
If your evals are the only quality system you have, you are mostly testing whether yesterday’s mistakes came back. Useful, but not enough.
What are the 12 failures your evals will never catch?
1. The agent solves the literal request, not the job
User says, “Add billing to this app.” The agent adds a checkout button. It passes a unit-ish eval for “billing integration.” But the real job included plan limits, trial state, webhooks, upgrade copy, and access control.
The agent did the noun. It missed the workflow.
Signal: follow-up turns full of “also” and “wait, this should…”
2. The user corrects the agent but the agent does not really absorb it
“No, not that table, the customer table.”
The agent apologizes, then keeps using the wrong table. This is conversation memory failing under normal human correction.
Signal: repeated corrections on the same entity, file, account, date, or goal.
3. The task completes but creates downstream cleanup
It writes the email. It summarizes the call. It updates the config. Then the user spends 25 minutes cleaning up tone, missed constraints, broken formatting, or tiny wrong assumptions.
Your eval says “accepted.” The user says nothing. The spreadsheet of resentment updates silently.
Signal: copy-edit bursts, manual rewrites, or user messages getting shorter after a long agent output.
^ calling it task completion because the agent produced 900 words nobody used
4. The agent asks the right question at the wrong time
Clarifying questions are good until they become friction.
If the user is in exploration mode, questions help. If the user is trying to get a boring operational task done, questions feel like the agent is making them manage a junior employee.
Most evals reward caution. Production users often punish it.
Signal: abandonment immediately after clarifying questions, especially on repetitive task categories.
5. The agent fails at “obvious from context”
The user assumes the agent knows the current project, the last import, the plan they are on, and which “Sam” they mean. Humans rely on context aggressively. Agents pretend to, then pick the wrong piece.
Signal: phrases like “we just talked about this,” “same as before,” “the other one,” and “not that Sam.”
6. The agent chooses a path that works once and breaks later
Evals usually check final output. They rarely check whether the route was robust.
The agent hardcodes a value. Uses a brittle selector. Calls a tool in an order that only works because the seed data is clean. Makes a fix that passes the current test but violates the product’s actual operating model.
Signal: the same task category comes back as support tickets 24-72 hours later.
7. The agent loses trust without losing usage
A user can keep using the product while trusting it less. They start giving smaller tasks. They add defensive instructions. They manually verify everything. They stop delegating the high-value work.
Your usage chart looks fine. Your product value is shrinking.
Signal: delegation scope gets narrower over time for the same user.
8. The agent fails when the user is already frustrated
If the user is calm, a partial answer is fine. If they are on their third try, a partial answer feels insulting. Evals do not measure emotional context. Production absolutely does.
Signal: user sentiment drops before task abandonment, not after.
9. The agent over-optimizes for politeness
It apologizes, hedges, explains, and wraps every action in polished mush. The answer is technically acceptable, but the experience feels slow and fake.
Signal: users interrupt long agent responses, ask for “just the answer,” or stop reading after verbose preambles.
10. The agent cannot recover from a bad first branch
Evals often start fresh. Production users do not. They drag the agent back from its own wrong turn, and many agents are terrible at being redirected.
Signal: session length increases while resolution probability decreases.
11. The agent passes known edge cases and fails boring normal cases
Teams love fancy evals. Adversarial prompts. Injection tests. Weird logic puzzles. All useful. But a shocking amount of production pain is boring: bad timezone handling, wrong account selected, output too long for the field, formatting the user has to redo every time.
Signal: high frequency of low-drama fixes inside otherwise “successful” sessions.
12. The agent succeeds but the user does not feel progress
The agent may be doing real work, but if the user cannot tell what changed, what remains, and what to do next, they leave with anxiety instead of confidence.
Agents need to create forward motion, not just output.
Signal: users ask status questions after completion: “so is it done?”, “what changed?”, “what should I do now?”
How should teams measure what evals miss?
Do not throw away evals. Add production conversation analysis next to them.
| Layer | Question it answers | Failure it catches |
|---|---|---|
| Evals | Did the agent handle known cases? | Regressions, prompt bugs, expected edge cases |
| Traces | What did the agent do internally? | Bad tool paths, loops, routing mistakes |
| Conversations | Did the user get what they needed? | Trust loss, correction loops, abandonment, cleanup |
| Outcomes | Did the work survive reality? | Reopens, reversals, refunds, churn |
The important move is connecting these layers. A failed conversation should become a new eval when the pattern is repeatable. A weird trace should be tied to the user turn that caused it.
At Agnost, this is the layer we care about most: the conversation signal that tells you when the agent is becoming less trusted. The logs already exist. The hard part is reading them like product data.
What is a practical workflow?
Pick 100 recent production conversations from your highest-value user segment. Not random traffic. High-value users. People whose trust matters.
Tag four things:
| Tag | What to look for |
|---|---|
| Correction | User had to restate, reverse, or clarify after agent action |
| Cleanup | User edited or constrained the output heavily |
| Abandonment | Session stopped without resolution |
| Trust loss | User narrowed scope, hedged, or stopped delegating |
Then ask the uncomfortable question: which of these failures would your eval suite have caught? Usually the answer is “almost none.” That is not an indictment. It is a map.
Turn the repeatable ones into evals. Keep the messy, contextual ones as production signals. Product teams should be in that review, not only ML engineers.
FAQ
Are evals still worth building?
Yes. Evals are your regression harness. They stop old mistakes from coming back. They are just not a full quality system for a live agent product.
How many production conversations should we review?
Enough to see patterns by segment. For an early product, 50-100 serious sessions per week is often plenty.
Should every production failure become an eval?
No. Only repeatable failures with a clear expected behavior. Some failures are better tracked as product metrics: correction rate, abandonment after clarification, cleanup burden, and delegation scope.
What is the best leading indicator?
Trust loss. Smaller tasks after a bad session are future churn warming up.
TL;DR: Evals catch known mistakes. Production catches the weird, contextual, trust-destroying failures that happen when real users delegate real work. Keep evals, but add conversation-level analysis for correction loops, cleanup burden, abandonment, and trust loss. That is where the failures that matter usually live.
Reading Time: ~9 min