---
title: "Turn Production Agent Failures Into Regression Tests"
description: "Build regression cases from production failures without replaying real side effects. Includes a dataset contract, baseline comparison, and release criteria."
author: "Agnost AI Team"
datePublished: "2026-10-10T00:00:00.000Z"
dateModified: "2026-10-29T00:00:00.000Z"
canonical: "https://agnost.ai/blog/production-failures-regression-tests/"
tags: ["Agent Evals","Agent Monitoring","Testing"]
---

# Turn Production Agent Failures Into Regression Tests

Build regression cases from production failures without replaying real side effects. Includes a dataset contract, baseline comparison, and release criteria.

**Turn a production failure into a regression test by preserving the inputs, the evidence available at the time, and the expected behavior.** Replay against isolated tools, compare the existing agent with the candidate, and include healthy cases that the change could break.

A transcript is a starting point. A regression case also needs a reproducible environment and an assertion that expresses the failure precisely.

![Regression workflow: capture evidence, freeze tool fixtures, compare baseline and candidate, then review regressions.](/blog/october-2026/regression-loop.svg)

## Capture the decision boundary

Suppose the agent retrieved a 14-day return policy but answered “30 days.” Save the exact user request and the retrieved policy version. Do not quietly replace the source with today's document: that would test a different situation.

For a false refund claim, save the requested order, the relevant authorization state, and the tool outcome. Remove personal information while preserving relationships between records. Two order IDs that must differ in the test must still differ after redaction.

A useful case has these fields:

```json
{
  "case_id": "return-window-001",
  "input": "How long do I have to return this order?",
  "tool_fixture": {"policy_version": "v2", "return_days": 14},
  "expected": {"return_days": 14, "must_not_claim_refund": true},
  "failure_type": "source_contradiction",
  "split": "regression"
}
```

This is a proposed test format, not an Agnost import schema. Adapt it to the runner you use.

## Freeze evidence, isolate side effects

Replace payment, email, account deletion, and ticket creation tools with sandbox implementations or recorded responses. A replay should not charge a customer again merely because the original run contained a payment request.

Recorded responses work when the candidate asks the same meaningful question. If it changes the query or tool arguments, a mock that returns the old answer regardless can hide a bug. Make fixtures validate expected inputs and fail on unrecognized operations. Record these failures as test infrastructure errors rather than incorrect model answers.

Keep a separate environment test for the real tool integration. A mocked replay can verify decision behavior while missing authentication, network, and provider-schema problems.

## Compare baseline and candidate case by case

Run both versions on the same dataset and tool contract. Save prompt, model configuration, application commit, evaluator version, and dataset version. For nondeterministic systems, repeat runs according to the decision's risk and report the number of repeats.

| Case outcome | Meaning | Review action |
| --- | --- | --- |
| Baseline fails, candidate passes | Intended improvement | Inspect the evidence, not only the aggregate score |
| Both pass | Preserved behavior | Keep representative controls |
| Baseline passes, candidate fails | Regression | Investigate before rollout |
| Both fail | Unresolved problem | Do not count it as a fix |
| Fixture or evaluator fails | Invalid measurement | Repair the test and rerun |

A useful result says which cases changed. “Average score increased” can hide a new severe failure behind improvements to easier cases.

## Keep a held-out check

Use the original incident while developing the fix, but hold back related cases. For the return-window example, include a different policy window, a missing policy, and a conflicting-document case. Otherwise a prompt that always says “14 days” can appear to solve retrieval grounding.

Human reviewers should agree on the expected behavior before looking at the candidate output. If the policy itself is ambiguous, mark the case for resolution instead of forcing a misleading pass/fail label.

The official [LangSmith evaluation guide](https://docs.langchain.com/langsmith/evaluation-types) distinguishes offline testing from live evaluation. Our recommendation is to keep both: replay establishes behavior on a controlled set, while production monitoring checks what happens after release.

## Make the test part of the change review

Attach the failing case, the candidate diff, and the baseline/candidate report to the pull request. Define release criteria before choosing the winning version. A practical gate might require every critical action-integrity test to pass and no new failures in a fixed control set. Those thresholds are a product decision, not a universal benchmark.

Agnost can help locate the conversations behind a recurring finding and track [proposed improvements](https://docs.agnost.ai/using-improvements). Your repository remains responsible for its executable tests and release checks.

Our [synthetic detector lab](/blog/agent-monitoring-detection-benchmark/) supplies a small downloadable starting set. For the earlier step of identifying a misleading completion claim, use [action verification](/blog/detect-ai-agent-false-success/).

## Step-by-step replay validation

Use this sequence when the release question is “did we fix the failure without breaking what already worked?”

### 1. Select representative historical runs

Choose cases across four buckets:

- the exact confirmed incident;
- semantically similar failures from the same cluster;
- nearby edge cases with different entities, policy versions or tool outcomes;
- healthy controls using the same tools and decision path.

Do not select only the most dramatic failures. Preserve the observed distribution in a separate analysis set, then add a small critical set that every release must pass.

### 2. Freeze inputs and external state

Version the user input, system and developer instructions, tool schemas, retrieval documents, business state and expected behavior. A run replayed against today’s policy is not a replay of an incident caused by last month’s policy.

### 3. Execute baseline and candidate

Run both versions with the same fixture contract. Save application commit, prompt version, model identifier and parameters, evaluator version and randomization settings. If the provider does not expose a stable model snapshot, record that limitation.

```bash
agent-eval run --dataset production-regressions.jsonl --variant deployed
agent-eval run --dataset production-regressions.jsonl --variant candidate
agent-eval compare --baseline deployed --candidate candidate
```

These commands illustrate the interface a runner should provide; they are not Agnost CLI commands.

### 4. Compare behavior and uncertainty

For deterministic invariants, require exact results. For model-judged qualities, report the rubric, judge version, repeated-run count and disagreement. Never convert a subjective score difference into certainty without calibration.

### 5. Add the confirmed failure permanently

Once reviewers agree on expected behavior, add the case to the regression set with its provenance and redaction record. Link it to the production finding and the pull request that introduced the fix.

## Safe rollout after replay

Replay reduces risk; it does not recreate every production condition. Use a staged rollout for meaningful agent changes.

| Stage | Recommended check | Example rollback signal |
| --- | --- | --- |
| Shadow | Candidate observes copied inputs without side effects | Candidate violates a critical invariant |
| Internal | Staff or test tenants use the candidate | Confirmed high-severity failure |
| Canary | Small eligible production cohort | Failure rate exceeds baseline guardrail |
| Expanded | Increase cohort after minimum evidence | Healthy-control regression or cost spike |
| Full | Continue monitoring by release | Recurrence of the original failure cluster |

Define the eligible denominator, minimum sample and rollback owner before rollout. Avoid a universal “5% canary” rule: a low-volume money-moving workflow and a high-volume search assistant have different risk profiles.

## Pull-request checklist

- Production finding and affected interaction IDs are linked.
- Personal data is redacted from fixtures.
- Side-effecting tools are isolated or sandboxed.
- Baseline and candidate use the same dataset contract.
- Critical failures pass.
- Healthy controls do not regress.
- Incomplete evidence is reported separately.
- Evaluator and dataset versions are recorded.
- Rollout cohort and rollback criteria are defined.
- Post-deployment recurrence monitoring has an owner.

See [From Production Failures to Pull Requests](/blog/turn-conversations-into-pull-requests/) for the surrounding engineering workflow.