Your eval suite passing is not the same thing as your agent working.
It means the agent handled a curated set of examples under controlled conditions. That is useful. It is also much narrower than the world your users live in. Production is messy. Users ask half-formed questions, change goals mid-thread, reference account state the model cannot see, and disappear without telling you why.
This is how an agent can pass every eval and still fail the business.
The failure does not look like a red error. It looks like a user who does not upgrade. A customer who stops coming back. A support ticket that never gets filed. A workflow that technically completed but took so much effort the user lost trust.
Why evals pass while users fail
Offline evals usually have three simplifying assumptions:
- The input is known.
- The success condition is known.
- The context is stable.
Production breaks all three.
The input is not known because users do not speak in your taxonomy. They do not say “I am experiencing setup friction on the OAuth integration.” They say “why is this thing still asking for permissions” or “nevermind I will do it manually.”
The success condition is not always known because the user’s goal is often implicit. A user asking “can this connect to HubSpot?” might be asking about an integration, a security review, a buying decision, or a migration blocker. The right answer depends on what they are trying to do next.
The context is not stable because production changes constantly. Your docs changed yesterday. A tool call degraded this morning. A customer imported weird data. A competitor launched a feature and now users ask different comparison questions.
Your eval suite can only test the world it was given. Production keeps creating new worlds.
The four production failures evals usually miss
Most teams think “agent failure” means the model said something wrong. That is only one category. The more expensive failures are subtler.
| Failure mode | Why evals miss it | What it costs |
|---|---|---|
| Silent non-resolution | The answer sounds plausible, but the user never reaches the goal | Churn, failed activation |
| Repeated friction | The same user hits the same wall across sessions | Trust decay |
| Misread buying intent | Upgrade or expansion questions get handled as generic support | Lost revenue |
| Workaround drift | Users start asking how to bypass the product | Retention risk |
The common thread: you need production behavior to see them. A single-turn eval cannot tell you whether the user came back tomorrow with the same unresolved problem. A test case cannot tell you that the agent answered correctly but still failed to move the user forward.
Silent non-resolution
This is the most common one.
The agent gives an answer. The answer is not obviously wrong. The conversation ends. Everyone moves on.
But the user did not get what they wanted. They stopped because the agent exhausted them, not because the problem was solved.
You can see this in production when users ask the same intent again later, when they abandon immediately after a response, or when they switch from goal language to workaround language. You will not see it in an eval that only scores the response text.
Repeated friction
A single failure is forgivable. Repetition breaks trust.
If a user asks about the same setup step on Monday and Wednesday, that is not two isolated chats. That is one unresolved product problem. Most eval suites cannot connect those dots because they are not user-level systems. They test examples, not relationships over time.
Production monitoring has to group conversations by user, account, and intent. Otherwise the most important word in agent reliability, “again,” disappears.
Misread buying intent
AI products lose money when agents fail sales-shaped conversations.
A user asks about limits, integrations, procurement, exports, security, or pricing. The agent treats it as informational support. The user was actually evaluating whether to buy, upgrade, or expand. That moment never becomes a conversion event, so your funnel says “no intent.”
The conversation says the opposite.
This is where production data is brutally useful. It surfaces the buying questions that did not convert, which are often more useful than the ones that did. The miss tells you what blocked the deal.
Workaround drift
When users stop asking “can the agent do this?” and start asking “can I do this another way?”, you are watching trust leave the product.
“Can I export this manually?”
“Is there a way to edit the file directly?”
“Can I skip this step?”
Those are not neutral support questions. They are signs that the product has stopped being the path of least resistance. If enough users ask them, you have an agent reliability problem and maybe a product problem.
What to measure instead of just eval pass rate
Keep your eval pass rate. Just stop treating it as the top-line health metric.
Add production metrics that answer whether users are actually succeeding:
| Metric | What it tells you |
|---|---|
| Unresolved intent rate | Share of conversations where the user’s goal was not met |
| Repeated unresolved intent | Same user or account hitting the same failure again |
| Frustration plus failure | Emotional friction attached to a concrete unresolved goal |
| Upgrade-intent miss rate | Buying conversations that did not progress |
| Post-change failure trend | Whether a shipped fix reduced failures in live traffic |
The phrase “in live traffic” matters. You cannot know whether the agent improved by watching the playground. You know by watching the next cohort of users who hit the same intent after the change ships.
The loop that actually improves an agent
The operating loop should look like this:
- Capture every production conversation.
- Classify the user’s intent.
- Detect whether the intent was resolved.
- Rank recurring failures by user and revenue impact.
- Fix the prompt, harness, tool, or config.
- Add the failure to evals so it does not regress.
- Watch production to confirm the failure rate drops.
This is the loop Agnost AI automates. It reads real conversations, finds the failure categories your evals did not know about, and turns high-impact patterns into reviewed pull requests against prompts, agent harnesses, and configs. The point is not to replace evals. The point is to give evals a live source of truth.
The uncomfortable truth
If your agent is already in production and your only quality number is eval pass rate, you do not know if the agent works. You know it handles the cases you wrote down.
That is a start.
But users do not churn on your written-down cases. They churn on the messy cases nobody noticed until production made them expensive.
FAQ
Should eval pass rate be a reliability KPI? Yes, but only as a regression KPI. It tells you whether known cases still work. It does not tell you whether users are succeeding in production.
How often should production failures become evals? Continuously. Any recurring production failure that gets fixed should become part of the eval suite, especially if it affected activation, churn, upgrade, or trust.
What is the fastest way to find production failures? Start with repeated unresolved intents. Find users or accounts who asked for the same thing more than once and still did not complete the goal. That is usually where the most painful failures are hiding.
Passing evals is table stakes. Catching production failures is the work that keeps users.
Agnost AI monitors production conversations, catches the agent failures your evals miss, and helps turn those patterns into fixes your team can review and ship.