One person says, “We need better evals.” Another says, “We need traces.” Product says, “Can we read what users are saying?” Everyone is right, which is annoying and very on-brand.
Evals, traces, and conversations are not competing approaches. They answer different questions. An eval is a controlled test of a known case. A trace is the internal record of what the agent did. A conversation shows whether the person got what they needed.
If you only have one of the three, you are looking through a keyhole.
^ discovering that passing evals, clean traces, and happy users are three separate things
What do evals catch?
Evals catch known failures and regressions.
They are best when you can say: given this input, the agent should do X, not Y. That makes evals great for prompt changes, model swaps, tool schema updates, and behavior you never want to break.
Good eval cases:
| Eval target | Example |
|---|---|
| Known regression | Agent must not invent a refund policy |
| Required behavior | Agent must ask for confirmation before deleting data |
| Classification | Agent must identify upgrade intent |
| Tool selection | Agent must use account lookup before answering billing status |
| Output constraint | Agent must return JSON that matches schema |
This is real value. Without evals, your team will keep rediscovering old mistakes and calling it learning. But evals have a ceiling.
They are usually cleaner than production. They freeze context. They compress messy behavior into one neat prompt. And because they need expected outputs, they bias toward failures that are easy to define.
Evals catch the known knowns.
They do not catch the user who asks for the same thing three ways because the agent keeps half-solving it. They do not catch trust narrowing. They do not catch a tool path that “works” but takes 28 steps.
^ the eval suite after your agent passes every test and still annoys customers in production
What do traces catch?
Traces catch agent process: prompts, model calls, retrieval, tool calls, retries, routing decisions, latency, token usage, and errors.
This is where engineering teams breathe easier because traces look like something we already understand. Distributed systems had traces. Agents have traces. Great. Mostly.
Traces are excellent for:
| Trace question | Why it matters |
|---|---|
| Which tool did the agent call? | Wrong tool choice causes wrong outcomes |
| How many steps did it take? | Long paths are expensive and fragile |
| Where did latency come from? | Slow agents lose users |
| Did retrieval bring the right context? | Bad context poisons answers |
| Did a tool error happen? | Infra failures still matter |
| Did the agent retry the same plan? | Loops burn cost and patience |
If an eval says the agent failed, traces explain how. If production gets weird, traces help engineers find the break. But traces do not tell you whether the user was satisfied.
A trace is the agent’s autobiography. Useful, but biased.
The agent can have a beautiful trace and still be wrong about the job. It can make all the right internal moves toward a bad interpretation of the user’s goal.
Traces catch the how.
They do not always catch the whether.
What do conversations catch?
Conversations catch user reality. This is the layer most teams underuse because it is messy: typos, frustration, vague goals, changed minds, dead stops, partial acceptance, and the occasional “bro what.”
That mess is the product.
Conversation analysis answers:
| Conversation question | Signal |
|---|---|
| Did the user repeat themselves? | Misunderstanding or ignored context |
| Did they correct the agent? | Scope or factual failure |
| Did they abandon after a response? | Possible unresolved task |
| Did they narrow future tasks? | Trust loss |
| Did they ask “is it done?” | Poor progress communication |
| Did they manually rewrite output? | Low usability |
| Did they come back with the same issue? | Failure survived the session |
This is where agent experience lives: in the pattern between what the user wanted, what the agent did, and how the user reacted.
Conversations are not as neat as evals or traces. You need clustering, tagging, examples, and product judgment. You need patterns, not one spicy transcript.
At Agnost, this is the gap we see constantly: teams have evals for correctness and traces for debugging, but no durable layer for user reaction. So they can answer “did the agent call the tool?” and “did the answer match the test?” but not “are users trusting it more this month?”
That last question is usually the one that matters.
Which layer should you use for which problem?
Cheat sheet:
| Problem | Best first layer | Why |
|---|---|---|
| Prompt regression after a change | Evals | You need repeatable checks |
| Tool calling bug | Traces | You need the internal path |
| Slow response | Traces | You need latency breakdown |
| User says “that is not what I meant” | Conversations | The issue is intent fit |
| Users churn after “successful” sessions | Conversations | Success metric is lying |
| Agent loops through retries | Traces plus conversations | Trace shows loop, conversation shows cost |
| New user intent appears | Conversations | Evals cannot test what you have not discovered |
| Model upgrade risk | Evals plus traces | Check behavior and process |
| Revenue leak or missed upgrade | Conversations | The signal is in user intent and timing |
A conversation pattern reveals that users repeatedly correct billing answers. Traces show the agent skipped account lookup when plan state was cached. Evals get updated so the agent must call account lookup before billing recommendations. Then production conversations are monitored to see whether correction rate drops.
That is a loop. Most teams have pieces. Few have the loop.
What does a healthy agent quality stack look like?
A sane stack has three review cadences.
| Cadence | Layer | Owner | Output |
|---|---|---|---|
| Every PR or prompt change | Evals | Engineering | Pass/fail gates |
| Daily or on incident | Traces | Engineering / ops | Root cause and debugging |
| Weekly | Conversations | Product / founder / engineering | User patterns and fixes |
Conversations are not just a support thing. Product needs them because roadmap signal hides there. Engineering needs them because invisible bugs surface there. Founders need them because that is where you see whether the agent is useful or just busy.
If eval failures never connect to production examples, they become academic. If traces never connect to user outcomes, they become expensive logs. If conversation reviews never connect to fixes, they become therapy.
You want each layer feeding the next.
^ the quality loop finally clicking: conversation pattern, trace root cause, eval guardrail, production check
How do you start if you have none of this?
Start with the most painful layer you are missing.
If every release breaks something old, build evals.
If failures are impossible to debug, instrument traces.
If users churn and nobody knows why, analyze conversations.
For most production agent teams, the honest answer is the third one. They have some evals, even if rough. They have some traces, even if ugly. They do not have a structured view of where trust breaks.
The first useful conversation system is simple:
- Cluster sessions by user intent.
- Track correction, repetition, abandonment, and clarification.
- Review the top failing intents weekly.
- Attach each fix to a production signal.
- Promote repeatable failures into evals.
This is not glamorous. It works.
The goal is not a giant dashboard. The goal is making the next product decision from what users are experiencing, not only what the test suite and trace viewer can see.
FAQ
Are traces better than evals?
No. They answer different questions. Evals tell you whether known behavior still works. Traces tell you what the agent did internally. You usually need both.
Can conversations replace evals?
No. Conversations discover production patterns. Evals turn repeatable patterns into guardrails so they do not come back.
What should founders look at first?
Look at conversations by high-value segment. Find where trust breaks, then use traces and evals to fix the repeatable issues.
What is the most ignored layer?
Conversations. Not raw transcripts as a hobby, but structured conversation analytics: intent clusters, failure signals, trust changes, and outcomes.
TL;DR: Evals catch known regressions. Traces catch the agent’s internal path. Conversations catch whether users got what they needed. You need all three, but the biggest blind spot is the conversation layer, where trust, intent, abandonment, and product value show up.
Reading Time: ~8 min