What Are Online Evals? Online vs Offline Evaluation for AI Agents

Online evals are evaluations that run on an AI agent’s real production traffic. Instead of feeding the agent a prepared test set, an evaluator (an LLM judge, a code check, or a user signal) scores the conversations, traces, or tool calls that real users already produced. Offline evals tell you whether a change is safe to ship. Online evals tell you whether the shipped agent is working.

The term comes from machine learning, where “offline” evaluation uses a held-out dataset and “online” evaluation measures a model while it serves traffic. For LLM agents, the split matters more than it did for classic models. An agent’s inputs are open-ended, its tools change state in other systems, and its failures are often fluent and polite. A test set can only contain the failures someone already imagined.

Online vs offline evals at a glance

	Offline evals	Online evals
Input	A fixed dataset of prepared cases	Real production conversations, traces, or spans
When it runs	Before release, usually in CI or an experiment	After the interaction, continuously
Ground truth	Expected answers, references, or a rubric	Usually none; a judge, a rule, or an outcome signal
Main question	Did this change make known cases worse?	What is going wrong for real users right now?
Catches	Regressions on known cases	Unknown failure modes, drift, new user goals
Misses	Anything not in the dataset	Nothing until it happens at least once
Cost driver	Dataset size times the number of runs	Traffic volume times judge cost, reduced by sampling
Output	A pass rate or score comparison between versions	Rates over time, flagged examples, alerts

Neither column replaces the other. The useful pattern, covered below, is a loop in which online evals find failures and those failures become offline test cases.

What an online eval is made of

Every online eval has the same five parts, whatever tool runs it.

1. A unit. What gets judged: a single LLM call (span), one request through the agent (trace), or a whole multi-turn conversation (session or thread). The unit decides which failures are even visible. A span-level judge cannot see that the user asked the same question four times. Read conversation-level vs trace-level online evals for how to choose.
2. A trigger. When the judgment runs. Span and trace evals can run as soon as the data arrives. Conversation evals usually wait until the conversation has gone quiet, because judging a half-finished conversation produces false failures.
3. An evaluator. The thing that decides. An LLM judge reading a rubric, deterministic code (schema validation, regex, a tool-result comparison), or a signal from the user or a downstream system, such as a thumbs-down, a refund request, or a support ticket.
4. A population. Which traffic is judged: everything, a sample, or a filtered slice such as one agent, one customer tier, or conversations that used a refund tool.
5. An action. What happens with the result: a dashboard rate, a review queue, an alert, a new regression test, or a code change.

Teams that skip part 5 end up with a dashboard of scores nobody acts on. The setup guide walks through each part with defaults.

What online evals catch that offline evals miss

Offline evals are strongest on known behavior: the canonical refund question, the formatting contract, the tool the agent must call. Production adds the cases nobody wrote down.

• New user goals. People ask an onboarding agent about billing, or ask a coding agent to explain a stack trace it never saw. No test case exists because nobody predicted the request.
• Silent wrong answers. The agent states a 30-day return window after retrieving a 14-day policy. Nothing errors. See how to catch wrong answers that don’t show up as errors.
• False success. The agent says the cancellation is done, but no cancellation call succeeded. See detecting when an agent claims success without acting.
• Multi-turn friction. Each reply is reasonable in isolation, but the user rephrases four times and leaves. Only a conversation-level evaluator sees the pattern.
• Drift. A model update, a prompt edit, or a changed upstream API slowly shifts behavior. Offline evals pass because the dataset did not change. See agent drift in production.

The 12 production failures your evals will never catch lists more. Read that title as “your offline evals”: every item on it is a job for an online eval.

What online evals cannot do

Online evals have limits worth stating plainly.

• They are retrospective. A failure has to happen before it can be scored. If a failure must never reach a user, it needs a guardrail or a code check in the request path, not an online eval. See online evals vs guardrails vs monitoring.
• They inherit the evidence you send. A judge cannot verify a refund it cannot see. If the tool result or the system-of-record outcome is not captured, the evaluator can only judge what the agent said.
• Judges make mistakes. An LLM judge needs a labeled sample before its rate is trustworthy. See how to calibrate an LLM judge.
• They cost money at scale. Judging every conversation with a strong model adds up. See what LLM-as-a-judge costs in production.

The loop: online failures become offline tests

The two kinds of eval are most useful when connected.

1. An online eval flags a pattern in production, for example the agent confirming refunds before the refund tool succeeds.
2. Someone reviews the flagged examples and confirms the failure is real.
3. A fix is proposed and reviewed: a prompt change, a tool contract, a code check.
4. Representative failing examples, plus healthy controls, become offline regression cases. Turn production failures into regression tests covers how to do this without replaying real side effects.
5. After the fix ships, the same online eval measures whether the failure rate actually fell.

Step 5 is the one most teams skip, and it is the one that tells you whether the fix worked. What to do when an online eval fails covers the whole path from flag to verified fix.

How Agnost AI runs online evals

Agnost is built around the online half of this loop, for conversational agents.

• Rules in plain English. You write the behavior you expect, such as “state the cancellation policy before completing a cancellation”. Each rule is an evaluator. How to define agent failures in natural language covers how to write rules a judge can apply consistently.
• Whole-conversation judgment. Once a conversation has been idle for 15 minutes, an LLM judge reads it end to end against your active rules and flags the specific assistant messages that broke one. Conversations are processed in batches, so this is a review layer, not a real-time filter.
• No sampling knob. Every analyzed conversation is judged, up to your plan’s monthly event limit. When you add or edit a rule, Agnost also scores your most recent 1,000 conversations so the rule starts with a baseline.
• Intents alongside rules. The same judge labels what users were trying to do, so a violation rate can be read per user goal.
• Alerts and fixes. A rule’s violation count can trigger a Slack or email alert. A fired alert can start Auto Improve, which opens a pull request for your team to review and re-checks the alert about an hour after the change merges.

What Agnost does not do is also worth knowing. Verdicts are pass/fail flags with the evidence message, not graded scores. There is no CI gate, and violations are a classified finding, not enforcement. Teams that need dataset experiments or a CI eval step often run Agnost alongside an offline eval tool. The online evaluation tools comparison shows where each product fits, and the Violations guide covers setup.

Where to start

If you run no online evals today, start small:

1. Pick one failure that would cost you a customer, such as a false completion, a skipped identity check, or a wrong policy answer.
2. Write it as a single, narrow rule with clear evidence.
3. Run it over a week of production conversations and read 20 flagged and 20 unflagged examples.
4. Fix the rule wording until you agree with the judge, then set an alert threshold.

One trusted evaluator that someone acts on is worth more than twenty scores on a dashboard. For the difference between the evaluation layers in more depth, read evals, traces, and conversations: what each one catches.

Frequently asked questions

What are online evals?

Online evals are evaluations that run on an AI system's real production traffic instead of a prepared test set. An evaluator, usually an LLM judge, a code check, or a user signal, scores live conversations, traces, or spans after they happen, so the team can see how the agent behaves with real users and real tools.

What is the difference between online and offline evals?

Offline evals run a fixed dataset through the agent before a change ships and compare the results with expected answers or a rubric. Online evals score production traffic after it happens. Offline evals answer whether a change is safe to release; online evals answer whether the released agent is actually working for users.

Do online evals replace offline evals?

No. Offline evals catch regressions on known cases before release. Online evals find the failures nobody wrote a test for. The useful loop turns online-eval failures into new offline test cases, so the same failure is caught before the next release.

Are online evals the same as guardrails?

No. A guardrail runs inline and can block or rewrite a response before the user sees it, so it must be fast and cheap. An online eval runs after the interaction and only observes, so it can use a slower, more careful judge and look at the whole conversation.

Should online evals score every conversation or a sample?

Score everything for rare, high-severity failures and when volume is low enough to afford it. Sample when the judge is expensive and the failure is common enough that a sample still finds it. Whatever you choose, record the denominator so a rate can be interpreted.

How does Agnost AI run online evals?

You write rules in plain English, such as 'confirm identity before sharing account details'. Agnost's LLM judge checks every analyzed production conversation against those rules once the conversation has been idle for 15 minutes, flags the assistant messages that broke a rule, reports violation counts, and can alert Slack or email when a threshold is crossed. An alert can also open a pull request that is re-checked after merge.