---
title: "Conversation-Level vs Trace-Level Online Evals: Which Failures Each Can See"
description: "Span and trace evals judge one model call or one request. Conversation-level evals judge the whole multi-turn session. Here is which agent failures each unit can and cannot see, and how to run both without double-counting."
author: "Agnost AI Team"
datePublished: "2026-11-07T00:00:00.000Z"
dateModified: "2026-11-07T00:00:00.000Z"
canonical: "https://agnost.ai/blog/conversation-level-vs-trace-level-online-evals/"
tags: ["Online Evals","Agent Evals","Conversation Analytics","Agent Monitoring"]
---

# Conversation-Level vs Trace-Level Online Evals: Which Failures Each Can See

Span and trace evals judge one model call or one request. Conversation-level evals judge the whole multi-turn session. Here is which agent failures each unit can and cannot see, and how to run both without double-counting.

**The unit you evaluate decides which failures you can see.** A span-level evaluator judges one model call. A trace-level evaluator judges one request through the agent, including its tool calls. A conversation-level evaluator judges the whole multi-turn session after it ends. Most agent failures that cost customers only exist at the conversation level, but most eval tooling started at the span.

This is not a question of which is better. Each unit catches failures the others cannot, and each has a cost. For the broader picture of online evaluation, start with [what are online evals](/blog/what-are-online-evals/).

## The three units

| Unit | What the judge reads | When it can run | Typical cost per judgment |
| --- | --- | --- | --- |
| **Span** | One LLM call: prompt, completion, maybe one tool call | Immediately | Low |
| **Trace** | One request: the user turn, every step and tool call, the final answer | When the request completes | Medium |
| **Conversation** (session, thread) | Every turn of the interaction, in order | After the conversation ends or goes idle | Higher |

Vendors use different names. LangSmith calls conversations "threads", Arize and PostHog call them "sessions", Opik uses "threads". The idea is the same.

## Failures only a conversation-level eval can see

Each of these passes every span and trace check:

- **Repetition and rephrasing.** The user asks the same thing three ways. Each answer is plausible on its own. See [repetition as a red flag](/blog/repetition-red-flag-looping-conversations/).
- **Skipped prerequisite.** The agent changes an account email in turn 6 but never verified identity in turns 1 to 5.
- **Contradiction across turns.** The agent quotes one price in turn 2 and another in turn 7.
- **Unresolved goal.** Every response is polite and accurate, and the user's actual request is never handled.
- **Silent abandonment.** The user stops replying after a confusing answer. There is no error and no thumbs-down. See [users who silently give up](/blog/silently-give-up-ai-products/).
- **Late false completion.** The agent says "done" in the last turn, and the action failed three turns earlier.

## Failures a trace-level eval catches better

Conversation-level judging is not a replacement for trace and span checks:

- **Wrong tool or bad arguments.** Visible inside one request. A trace check can point to the exact step.
- **Ungrounded answer.** The answer contradicts the document retrieved for that same request.
- **Format and schema errors.** Better handled by code at the span level.
- **Latency and cost outliers.** Per-step metrics, not judgments.
- **Duplicate tool calls.** See [detecting duplicate tool calls](/blog/detect-stop-duplicate-tool-calls/).

A conversation judge reading 30 turns can miss a subtle argument error in step 4 of turn 12. Narrow checks are more precise for narrow failures.

## The timing problem

A trace is complete when the request returns. A conversation has no clear end: users come back after lunch, or never reply.

Judging too early creates false failures. The identity check looks skipped because it happens next turn. The question looks unanswered because the answer is still coming. So conversation-level evaluators usually wait for inactivity, and the defaults vary widely. LangSmith's multi-turn evaluators wait until a thread has been idle for 10 minutes by default. Datadog treats a session as complete after 30 minutes of inactivity. Braintrust's trace and group scores wait 30 seconds by default, and Opik's thread rules use a configurable cooldown. Agnost waits for 15 minutes of inactivity before judging.

The trade-off is latency. A conversation-level eval cannot tell you about a failure while the conversation is still happening. If you need that, you need a guardrail in the request path, not an online eval. See [online evals vs guardrails vs monitoring](/blog/online-evals-vs-guardrails-vs-monitoring/).

## Choosing a unit for each evaluator

| Question you are asking | Unit |
| --- | --- |
| Is this output valid JSON? | Span (code) |
| Did the agent pick the right tool with the right arguments? | Trace |
| Does the answer match what was retrieved? | Trace |
| Did the user get what they came for? | Conversation |
| Was a required step done before the action? | Conversation |
| Did the agent contradict itself or loop? | Conversation |
| Did the agent claim success that didn't happen? | Conversation plus outcome event |

## Running both without double-counting

When the same failure can be flagged at several levels, rates become confusing. Three rules help:

1. **Assign each failure to one unit.** "Wrong refund amount" lives at the trace level; "refund never resolved" lives at the conversation level.
2. **Point conversation verdicts at a turn.** A conversation-level judge should return which message failed, so the finding links to the trace that produced it.
3. **Report conversation rates per conversation, trace rates per trace.** Mixing denominators makes a failure look ten times more or less common than it is.

## Where Agnost fits

Agnost runs its online evals at the conversation level. After a conversation has been idle for 15 minutes, an LLM judge reads it from start to finish against your plain-English rules and flags the specific assistant messages that broke one. That keeps the advantage of whole-conversation context while making every finding point to a turn you can open. Tool calls nested under a turn are left out of the judge's transcript and tracked separately, with success rates and error distributions. If a rule depends on a tool outcome, record that outcome as its own event.

If your team already runs span or trace evals in a tracing tool, the conversation layer is the gap to fill. The [online evaluation tools comparison](/blog/best-online-evaluation-tools-ai-agents/) shows which tools judge at which units, and [evals, traces, and conversations](/blog/evals-traces-conversations-what-each-catches/) covers how the layers fit together.

## Frequently asked questions

### What is a conversation-level online eval?

A conversation-level (also called session-level or thread-level) online eval judges an entire multi-turn interaction after it ends, rather than a single model call or request. It can see patterns across turns, such as a user repeating a question, a required step skipped several turns before an action, or a goal that was never resolved.

### When should I use trace-level evals instead?

Use trace-level evals for failures contained within one request: a wrong tool choice, bad tool arguments, an answer that contradicts the document retrieved in the same request, or a slow step. They are cheaper per judgment and can run as soon as the trace arrives.

### Why wait for a conversation to go idle before judging it?

Judging a conversation that is still in progress produces false failures: a step that looks skipped may happen two turns later, and a question that looks unanswered may be answered next. Most tools wait for a period of inactivity, often around 15 minutes, before running a conversation-level judge.

### Are conversation-level evals more expensive?

Per judgment, yes, because the judge reads more tokens. Per useful finding, often not, because one conversation-level judgment can replace several trace-level ones and catches failures they cannot see. Filtering to relevant conversations controls cost better than sampling.