---
title: "Your Online Eval Failed. Now What? From Flag to Verified Fix"
description: "An online eval flag is the start of the work, not the end. Here is the path from a failing production eval to a confirmed root cause, a reviewed fix, a regression test, and a measured before-and-after, with the mistakes that break each step."
author: "Agnost AI Team"
datePublished: "2026-11-07T00:00:00.000Z"
dateModified: "2026-11-07T00:00:00.000Z"
canonical: "https://agnost.ai/blog/online-eval-failed-what-next/"
tags: ["Online Evals","Agent Improvement","Self-Improving Agents","Agent Evals"]
---

# Your Online Eval Failed. Now What? From Flag to Verified Fix

An online eval flag is the start of the work, not the end. Here is the path from a failing production eval to a confirmed root cause, a reviewed fix, a regression test, and a measured before-and-after, with the mistakes that break each step.

**A failing online eval tells you something went wrong in production. It does not tell you why, whether it matters, or whether your fix worked.** Teams that treat the flag as the finish line end up with a dashboard of red numbers and the same failures next month. The useful path has six steps: confirm, check for change, find the cause, fix, lock it in, and measure.

If you have not set up online evals yet, start with [how to set up online evals in production](/blog/how-to-set-up-online-evals-production/).

## Step 1: Confirm the failure is real

Before anyone touches the agent, read the evidence.

- Open 20 flagged conversations. Do you agree with the judge on each one?
- Open 20 conversations the judge passed in the same slice. Are any of them failures it missed?
- Note any flag you disagree with, and why.

If you disagree on more than a few, the problem is the evaluator. Tighten the rubric, add the exception it missed, or add an "insufficient evidence" outcome. [How to calibrate an LLM judge](/blog/calibrate-llm-judge-agent-monitoring/) covers this properly.

## Step 2: Check what changed

A rate that moves suddenly usually has a specific cause. Check, in order:

1. **The evaluator.** A rubric edit, a new judge model, or a changed filter. If any of these changed, the rate is not comparable to last week.
2. **The agent.** A prompt change, a model upgrade, a new tool, a changed tool schema.
3. **The world.** An upstream API that now returns different errors, a policy document that was updated, a product launch that brought new kinds of users.
4. **The traffic mix.** A spike from one customer or one feature can move the overall rate without any behavior change.

A rate that creeps up slowly is more likely drift. See [agent drift and production degradation](/blog/agent-drift-production-degradation/).

## Step 3: Group the failures by cause

Twenty failures rarely have twenty causes. Read the confirmed examples and sort them:

| Cause | What it looks like | Typical fix |
| --- | --- | --- |
| Prompt gap | The agent was never told the rule, or told ambiguously | Prompt or system-instruction change |
| Missing guardrail | The agent can claim an action it did not take | Code check between tool result and response |
| Tool error | The tool failed or returned a business error the agent ignored | Tool contract, error handling, retry policy |
| Retrieval | The right document was missing, stale, or outranked | Index, chunking, or ranking change |
| Missing feature | Users ask for something the agent cannot do | Product decision, or an honest refusal |
| Evaluator | The rubric flagged acceptable behavior | Rubric change |

Fix the largest group first. A fix that addresses 60% of failures today beats a perfect fix for a rare edge case. [How to prioritize which agent bugs to fix](/blog/prioritize-ai-agent-bugs-to-fix/) has a scoring method.

## Step 4: Ship the smallest fix that addresses the cause

Prefer the narrowest change that removes the cause:

- A missing rule belongs in the prompt.
- A claim that must be true belongs in code, not in a prompt. "Never say a refund is complete unless the refund tool succeeded" is more reliable as a check than as an instruction.
- A tool that fails silently needs its contract fixed, not a prompt that asks the agent to be careful.

Write down which eval should move, and in which direction, before you ship. That makes step 6 a check instead of a story.

## Step 5: Lock it in with regression cases

Take representative failing conversations and turn them into offline test cases, with healthy controls next to them so a fix that over-corrects is caught too. [Turn production failures into regression tests](/blog/production-failures-regression-tests/) covers the dataset contract and how to avoid replaying real side effects. This is how the online eval feeds the offline one: next release, the same failure is caught before users see it.

## Step 6: Measure the same eval after the fix

Compare the failure rate before and after, using:

- the same rubric version and judge model;
- comparable traffic (same weekday pattern, no launch in one window and not the other);
- the denominator, not only the count;
- enough judged conversations that the difference is not noise.

Record one of four outcomes: improved, unchanged, regressed, or inconclusive. "Inconclusive" is a legitimate answer and better than claiming a win from three conversations. [How to measure AI agent improvement](/blog/measure-ai-agent-improvement/) goes deeper on baselines.

## Where teams lose the thread

- **No owner.** The alert goes to a channel where everyone assumes someone else is looking.
- **Fixing the judge silently.** The rubric is relaxed until the number turns green, and nobody records that the evaluator changed.
- **No before-and-after.** The fix ships, the team moves on, and nobody checks whether the rate fell.
- **One-off fixes.** The failure is fixed but never becomes a regression case, so the next prompt rewrite brings it back.

## How Agnost AI shortens the loop

Agnost connects the online eval to the fix and the measurement. A plain-English rule flags the assistant messages that broke it, with the conversation as evidence. A Slack or email alert fires when the violation count crosses a threshold. With Auto Improve connected, that alert starts a run that gathers the evidence, reads your repository, names a root-cause category, and opens a pull request for your team to review. Agnost never merges it, skips a run when a pull request for the same alert is still open, and caps runs per day and per alert.

About an hour after the pull request merges, Agnost re-checks the alert condition and records the outcome as improved, unchanged, regressed, or inconclusive. You still confirm the root cause, review the diff, and run your CI. The difference is that steps 3, 4, and 6 start for you. The [Improvements guide](https://docs.agnost.ai/using-improvements) covers setup, and [from dashboards to pull requests](/blog/dashboards-to-pull-requests-closing-the-loop/) explains the idea behind it.

## Frequently asked questions

### What should I do when an online eval starts failing?

Read a sample of flagged conversations to confirm the failure is real, check whether the agent or the evaluator changed, group the failures by cause, ship the smallest fix that addresses the most common cause, add regression cases from the failing examples, and compare the same online eval's rate before and after the fix.

### How do I know whether the eval or the agent is wrong?

Check what changed first: the agent version, the model, the prompt, the tools, or the rubric and judge model. Then read 20 flagged and 20 unflagged conversations. If you disagree with the judge on several of them, fix the rubric before touching the agent.

### How do I prove a fix worked?

Measure the same online eval, with the same rubric and judge, on comparable traffic before and after the change. Report the rate with its denominator. A drop after a rubric edit, or on a week with very different traffic, is not evidence the agent improved.

### Can online eval failures be fixed automatically?

Tools can now propose fixes as pull requests from production failures, including Agnost Auto Improve, LangSmith Engine, Arize Signal, and PostHog Self-driving. They shorten the path from flag to change, but a person still needs to confirm the root cause, review the diff, and check the measured outcome.