An online eval is only as good as the evidence it reads and the action it triggers. Most teams start by choosing a judge model. Start instead with the failure you care about, the evidence that proves it, and who will act when it fires. This guide walks through the setup in the order that avoids rework. For background on what online evals are, read what are online evals.
Step 1: Capture evidence a judge can use
A judge can only evaluate what it receives. Before writing any rubric, check that each conversation includes:
- every user and assistant message, in order, with timestamps;
- tool calls with their inputs, outputs, and errors, including business errors returned with an HTTP 200;
- retrieved documents or their identifiers and versions;
- the agent, prompt, and model version that produced each response;
- a stable conversation ID and, where possible, a user or account ID;
- outcome events from systems of record, such as “refund settled” or “ticket reopened”.
The last item is the one most often missing. Without it, a judge can only decide whether the agent claimed success. OpenTelemetry instrumentation covers most of this list; see how to instrument an AI agent with OpenTelemetry. Redact what you must before export, following how to redact agent traces, but keep enough structure for the judge to see what happened.
Step 2: Pick the unit for each failure
Match the evaluation unit to the failure, not to whatever your tool defaults to.
| Failure | Unit | Why |
|---|---|---|
| Malformed JSON, schema violation | Span | Visible in one model call; use code, not a judge |
| Wrong tool or wrong arguments | Trace | Needs the request and the tool call together |
| Answer contradicts the retrieved policy | Trace | Needs retrieval and answer in one view |
| User repeats themselves, gives up | Conversation | Only visible across turns |
| Required step skipped before an action (identity check, disclosure) | Conversation | The step and the action may be turns apart |
| Claimed completion without a successful action | Conversation plus outcome event | Needs the claim and the system-of-record result |
Conversation-level evaluation should wait until the conversation is over. A common default is to judge after a period of inactivity, often 15 minutes. Conversation-level vs trace-level online evals covers the trade-offs.
Step 3: Write one narrow rubric per failure
Broad rubrics such as “rate helpfulness from 1 to 10” produce numbers nobody can act on. Write one behavior per evaluator and make it pass/fail.
Name: refund-claimed-before-confirmation
Scope: conversations where the user asks for a refund.
Pass: the assistant describes the refund as complete only after a
refund tool result confirms it, or describes it as submitted/pending.
Fail: the assistant says the refund is done, issued, or processed
and no successful refund tool result precedes that message.
Exceptions: the user is told a human agent will process the refund.
Insufficient evidence: no refund tool calls were captured at all.
Output: PASS | FAIL | INSUFFICIENT_EVIDENCE, plus the message ID of
the failing assistant turn.Three details matter:
- An explicit “insufficient evidence” outcome. Forcing every case into pass or fail hides instrumentation gaps.
- Point at the evidence. Asking the judge to return the failing message ID makes every verdict quick to verify.
- Write the exceptions down. Most false positives come from legitimate cases the rubric did not anticipate.
How to define AI agent failures in natural language has more rewrites of vague rules into checkable ones.
Step 4: Use code where code works
An LLM judge is the expensive, fuzzy option. Use it only where the check needs judgment.
- Code: schema validity, required fields, forbidden strings, tool error codes, latency, retries, duplicate tool calls, matching a quoted number against a tool result.
- LLM judge: policy adherence, whether an answer is grounded in retrieved content, whether a required step happened in the right order, whether the user’s goal was met.
- User and system signals: thumbs-down, rephrasing, escalation requests, ticket reopened, refund reversed. These are noisy alone but useful for finding what to write a rubric for.
Step 5: Choose a population and sampling rate
| Situation | Population |
|---|---|
| Rare, severe failure (data exposure, unauthorized action) | Every conversation that could contain it |
| Common, low-severity failure (tone, verbosity) | A sample is enough |
| New rubric being calibrated | Every conversation for a short window, then decide |
| High volume and an expensive judge | Filter to the relevant slice first, then sample |
Filtering beats sampling. A refund rubric only needs conversations that touched refunds. Whatever you choose, store the number of conversations judged, not just the number flagged. A count without a denominator cannot be compared week to week. Sampling agent traces without hiding failures covers the traps, and what LLM-as-a-judge costs in production shows how to estimate the bill.
Step 6: Calibrate before you trust the rate
Before alerting on a judge, measure it.
- Pull 50 to 100 conversations, including some the judge flagged, some it passed, and some near misses.
- Have two people label them independently with the same rubric.
- Where the humans disagree, fix the rubric or the policy. A judge cannot be more consistent than its instructions.
- Compare the judge with the agreed labels. Look at false positives and false negatives separately; they cost different things.
- Re-run the comparison whenever you change the judge model or the prompt.
How to calibrate an LLM judge for agent monitoring covers this in detail.
Step 7: Route results to someone who will act
Decide the action before turning the evaluator on.
- Alert when a rate or count crosses a threshold over a window, with a minimum number of judged conversations. Never page on a single verdict.
- Include examples in the alert, so the recipient can confirm the failure in a minute instead of opening a dashboard.
- Match the channel to severity. Immediate for rare, harmful failures; a daily or weekly digest for the rest. A weekly agent review is a good home for low-severity trends.
- Close the loop. After a fix ships, compare the rate before and after. See what to do when an online eval fails.
Step 8: Version everything
Record, for each verdict, the rubric version, the judge model, and the agent version that produced the conversation. When a rate moves, you need to know whether the agent changed or the evaluator did. A lower failure rate after a rubric edit is not evidence the agent improved.
A starter set of online evals
For a support or task agent, these five evaluators cover most of the expensive failures:
- False completion. Claimed success without a successful action.
- Grounding. Factual claims contradict the retrieved policy or tool result.
- Required step. Identity check, disclosure, or confirmation skipped before an action.
- Escalation. The user asked for a human and did not get a transfer or a clear reason.
- Unresolved goal. The conversation ended without the user’s request being answered or handed off.
Doing this in Agnost AI
In Agnost, each rubric becomes a rule in the Violations view, written in plain English. Agnost judges every analyzed conversation against your active rules once it has been idle for 15 minutes, flags the assistant messages that broke a rule, and scores your most recent 1,000 conversations when you add or edit a rule, so you can read flagged examples before setting a threshold. From a rule you can create a Slack or email alert with a count threshold and lookback window, and a fired alert can start an Auto Improve pull request. The Online evals guide in the docs maps each step above to the product.
Frequently asked questions
How do I start running online evals on my AI agent?
Capture full conversations with tool inputs and results, pick one high-cost failure, write it as a narrow pass/fail rubric, run an LLM judge on a week of production traffic, compare its verdicts with 50 to 100 human labels, and only then add an alert threshold. Add more evaluators one at a time.
What should an online eval rubric look like?
One behavior per rubric, with scope, the required behavior, the evidence that proves a violation, explicit exceptions, and an 'insufficient evidence' outcome. Prefer pass/fail over a 1 to 10 score; numeric scales drift and are hard for reviewers to agree on.
Which model should I use as the judge?
Start with a strong model on a small sample to establish what correct judgments look like, then test whether a cheaper model agrees with those labels at an acceptable rate. Keep the judge model and prompt versioned, because changing either changes the rate.
How do I alert on online eval results without noise?
Alert on a rate or count over a window with a minimum sample size, not on single verdicts. Route rare, severe failures to an immediate channel and common, low-severity ones to a daily or weekly review. Include example conversations in the alert so the recipient can verify it quickly.