Evals are how you test the world you can imagine. Production is how you discover the world you cannot.
That is the gap most AI teams are living inside right now. They have an eval suite. It passes. The launch looks clean. Then real users arrive and the agent starts failing in ways nobody wrote a test for: a weird phrasing, a buried requirement, a multi-turn correction, a tool edge case, a customer using the product in a way the team never expected.
The eval was not useless. It did exactly what it was designed to do. It checked the known unknowns. The problem is that most expensive agent failures are unknown unknowns, and unknown unknowns do not announce themselves in a spreadsheet before launch. They appear in production, inside real conversations, attached to real revenue.
What evals are actually good at
Evals are necessary. If you run an agent without evals, you are flying blind on the failures you already know exist. That is irresponsible.
The right eval suite catches things like:
| Evals are good for | Example |
|---|---|
| Known regressions | “Refund policy answer should not hallucinate a 30-day window” |
| Contract behavior | “Agent must call the pricing tool before answering enterprise pricing” |
| Safety boundaries | “Agent must not reveal internal prompts” |
| Common happy paths | “New user can connect Slack in under five turns” |
| Model or prompt comparisons | “Prompt B beats prompt A on this labeled set” |
That work matters. It prevents you from re-breaking things you already fixed. It gives your team a baseline before shipping. It catches obvious regressions before users do.
But notice the shape of the table. Every row starts with something you already know to test.
That is the core limitation. Evals cannot cover what nobody has noticed yet. They cannot tell you that a new customer segment asks about setup in a completely different vocabulary. They cannot tell you that your agent technically completes a task but leaves the user less confident than when they started. They cannot tell you that users who ask one specific question in onboarding are 3x less likely to convert seven days later.
Those are not eval problems. Those are production reality problems.
The known unknowns problem
A known unknown is a failure class you can name.
“The agent sometimes gives the wrong refund answer.”
“The billing flow breaks if the user asks about annual plans before connecting Stripe.”
“The model over-apologizes when the user reports a bug.”
Once you can name the failure, writing an eval is straightforward. You collect examples, define the expected behavior, run it on every change, and keep the system from sliding backward. This is good engineering.
The trap is when teams mistake the named failure set for the whole failure surface.
It never is. Your eval set is a map of past pain. It is not a map of future usage. The moment your agent meets real users, the surface area expands:
- New words for old intents
- Multi-step tasks that cross product boundaries
- Customers combining features in ways docs never described
- Ambiguous asks where the right answer depends on account context
- Frustrated users who do not clearly state what went wrong
- Silent failures where the user gives up without complaining
You do not get those by brainstorming in a room. You get them by watching production.
Unknown unknowns are where churn hides
The most damaging failures are often the least visible ones.
A user asks your onboarding agent how to import a CSV. The agent gives a technically correct answer, but it assumes a clean file. The user’s file has duplicate column names. They try again, get another generic answer, then leave. No thumbs down. No ticket. No error log. Your eval suite still passes.
From your dashboard, that looks like a normal non-conversion.
From the conversation, it is obvious: the agent failed to resolve setup friction. The user did not churn because they were unqualified. They churned because production exposed an edge case your evals never imagined.
This is why production conversation data matters so much. It contains the categories you did not know to track:
| Unknown unknown | What it looks like in production |
|---|---|
| Hidden setup friction | Users repeatedly ask about a step docs assume is obvious |
| Trust decay | Users start verifying every answer instead of accepting the agent’s output |
| Misclassified intent | Agent treats a cancellation risk as a generic support question |
| Silent non-resolution | Conversation ends politely but the user’s goal never happens |
| Workaround-seeking | User asks how to bypass the product because the agent failed |
| Revenue hesitation | User asks upgrade-shaped questions but never reaches the value moment |
None of these require the model to crash. That is the problem. The agent can look healthy from an engineering dashboard while quietly damaging the business.
Production should feed the eval suite
The answer is not “stop running evals.” The answer is to stop treating evals as the source of truth.
The best loop is:
- Watch real production conversations.
- Detect recurring failure patterns by intent.
- Turn the highest-impact patterns into fixes.
- Add representative examples back into evals so the fix does not regress.
- Keep watching production for the next unknown unknown.
In other words, evals should be downstream of production data, not a substitute for it.
If your eval suite is not being updated from real failures, it is aging. It might still pass, but it is passing against yesterday’s understanding of the product.
This is the part most teams miss. They have a static eval set written before launch, then a production system that changes every day: new users, new prompts, new tools, new product surfaces, new expectations. The eval suite stays stable while reality moves.
That stability feels comforting. It is actually dangerous.
How to discover the failures you did not know to test
You need a production layer that reads conversations as product data, not just logs.
At minimum, it should answer five questions:
| Question | Why it matters |
|---|---|
| What did the user actually want? | You cannot measure resolution without intent |
| Did the agent resolve it? | Completion matters more than answer quality in isolation |
| Where did the user show friction? | Frustration often appears before churn |
| Is this pattern repeating across accounts? | One weird chat is noise. A recurring intent is a roadmap item |
| Did a shipped fix reduce the failure rate? | Improvement has to be measured on live traffic |
This is what Agnost AI is built to do. It connects to your agent, reads production conversations, auto-generates the intents and failure categories that matter to your product, and surfaces the patterns your evals missed. When a recurring failure maps to a prompt, harness, or config issue, Agnost can turn it into a reviewable pull request.
The important shift is that your eval suite stops being a guess about what might fail and becomes a record of what actually failed.
A simple operating model
If you are running an AI agent in production, split your reliability work into two lanes:
| Lane | Purpose | Source of truth |
|---|---|---|
| Evals | Prevent known failures from coming back | Curated examples |
| Production monitoring | Discover unknown failures users are hitting now | Real conversations |
One lane protects you from regression. The other protects you from surprise.
You need both.
But if you only have evals, you have a system that is good at passing tests and weak at discovering reality. That might be fine for a demo. It is not enough for a product users depend on.
FAQ
Are evals still worth investing in? Yes. Evals are the right way to catch known regressions and enforce behavior you already understand. The mistake is expecting them to discover failure modes nobody has observed yet.
What is the biggest blind spot in most eval suites? They overrepresent clean, named examples and underrepresent messy multi-turn production behavior. Users rarely fail in the exact format your eval author imagined.
How should production data change my eval process? Every recurring production failure should either become a product fix, a prompt or harness change, or a new eval case. Production discovers the unknown unknown. Evals make sure it does not become a known regression later.
Your evals are necessary. They are not enough. If you want to catch the failures that actually cost money, you need to read what your users are doing in production.
Agnost AI catches the failures your evals miss by continuously monitoring production conversations, surfacing recurring failure patterns, and turning them into reviewed fixes.