How to compare task success before and after a change
- Tag every event with an agent version. Add an
agent_version(or prompt hash, or git SHA) to event metadata in the SDK, or as a custom span attribute over OpenTelemetry. The data governance guide showsagent_versionas a recommended metadata field. Without it you are comparing calendar dates, not agents. - Freeze the success definition before you change anything. Same rule, same eligible conversations, same window length on both sides. Define it per intent: “user wants to connect a data source” succeeds when the connection tool returns success and the user doesn’t ask again. If the definition changes with the agent, the comparison is meaningless. How to measure AI agent task success without inflating the number covers writing that definition.
- Snapshot the baseline. Measure the rate for each intent over, for example, the two weeks before the change. Record the denominator and the share of unknown outcomes, not only the percentage.
- Replay past failures offline first. Turn the failing conversations from the baseline into a test set and run the new version against it. This catches regressions before users see them; see turn production agent failures into regression tests.
- Ship, then compare per intent with a version filter. Measure the same rule on post-change traffic, filtered by
agent_version. If you can split live traffic between versions, do; it removes seasonality and traffic-mix effects. - Check every intent, not only the one you targeted. A prompt edit for billing can quietly break onboarding. Look for the regression you didn’t plan for.
- Require the change to hold. One good week can be noise. A second comparable window confirming the first is what makes it a result.
Why “it looks better in testing” isn’t enough
Offline test sets only check cases you already know about. You wrote the inputs and decided what correct looks like. That’s the right tool for catching regressions on known behavior, and you should keep it.
It can’t tell you that users who say “cancel” instead of “return” get routed to the wrong flow, or that pricing questions get a vague non-answer. Those failures live in production conversations your tests never imagined. So when someone says “the agent is better now,” ask: better at what, for whom, measured how? “Better on the 40 examples we picked three months ago” is overfitting to your own assumptions.
What to compare
An agent gets better in one way that matters: fewer users leave with their problem unsolved. Token counts and latency matter for cost and experience, but they don’t tell you whether the agent did its job. These do:
| Metric | What it tells you | Why it beats a static test set |
|---|---|---|
| Unresolved rate per intent | Share of conversations where the user’s goal wasn’t met | Counts real goals, not scripted ones |
| Friction rate | Conversations with rephrasing, corrections (“that’s not what I asked”), or abandonment | Surfaces failure modes you didn’t think to test |
| Escalation and fallback rate | How often the agent hands off or says it can’t help | A direct proxy for the agent giving up |
| Goal completion per intent | Whether users finished the downstream action (connected, upgraded, filed) | Ties the agent to product outcomes, not answer quality |
| Multi-turn drop-off | Where in the conversation users stop | Single-turn test sets miss it |
Do this per intent. “Overall quality went up 2%” is an average of a dozen intents moving in different directions: one improves, another regresses, the line stays flat, and you conclude nothing worked when half of it did.
An illustrative before-and-after
An onboarding agent’s setup intent is failing often: users ask about OAuth scopes and get a wall of text nobody finishes. You rewrite that part of the prompt to give one step at a time and confirm before moving on, tag the release agent_version: 2026-05-12, and merge.
| Cohort | Window | Setup failure rate | Median turns to completion |
|---|---|---|---|
| Before | 2 weeks before the merge | 38% | 11 |
| After, week 1 | Merge to +7 days | 24% | 7 |
| After, week 2 | +8 to +14 days | 21% | 6 |
Illustrative numbers, not customer data. The claim you can now make is specific: the change cut setup failures on that intent and shortened the path, and the second week confirms the first. That is a result you can defend; “the new prompt looks cleaner in the playground” is not.
Confounders to rule out
- Traffic mix. A marketing push or a new customer segment mid-window makes cohorts incomparable. Segment by source or plan using metadata.
- Seasonality. B2B agent traffic looks different on weekends and at month-end. Compare like windows.
- Wider blast radius than expected. One prompt section can shift behavior on unrelated intents, which is why step 6 exists.
- A moving definition. If an LLM judge decides success, keep its prompt and model fixed across both windows.
Offline experiments and production comparison answer different questions
| Offline experiment | Production before-and-after | |
|---|---|---|
| Question | Does the new version pass cases we know about? | Do real users succeed more often? |
| Runs | Before shipping | After shipping |
| Catches | Regressions on known behavior | Unknown failure modes, collateral damage |
| Needs | A dataset of past cases and scorers | Version tags, a fixed success rule, comparable traffic |
The strongest loop runs both: past failures become the offline dataset, the change ships only if it passes, and production confirms the effect. That loop is described in more detail in how to build a continuous improvement loop for LLM agents.
Tools that do this
- Agnost AI. Auto Improve investigates a recurring finding, tests the proposed change against past runs, opens a pull request, and shows outcome measurements after deployment next to the baseline evidence. Conversations can be filtered by metadata such as agent version, and intents and rules give you the per-intent success definition. It is not an offline experiment framework for large dataset suites.
- Braintrust. Select a baseline experiment and it aligns test cases across experiments, adds score deltas to each row, and marks improvements and regressions, with a summary grade per experiment. The strongest choice for eval-first offline comparison. Free Starter tier; Pro $249/month.
- LangSmith. A compare view over two or more experiments on a dataset, with a source experiment as baseline and runs that regressed or improved on each feedback key highlighted. The natural fit for LangChain and LangGraph teams. Free Developer seat; Plus $39/seat/month.
- Langfuse. Open-source (MIT core) datasets and experiments, with versioned datasets so you can test a new agent version against the same baseline dataset version. Free to self-host; cloud from $29/month.
- Opik. Open-source (Apache-2.0) online evaluation rules that score production threads, including a
SessionCompletenessQualityjudge for whether the user’s goals were met. Rules only score conversations logged after you create them, so set the rule up before the change to get a baseline.
For a wider comparison of measurement tools, see the best tools to measure AI agent task completion.
To start tagging versions, add agent_version to your event metadata following the quickstart.
Frequently asked questions
How do I know if my AI agent is actually getting better?
Compare the same outcome measure on real traffic before and after each change, per intent, over comparable windows. If the failure rate for the intent you targeted drops and holds for a second window, and no other intent got worse, the change helped. A cleaner demo is not evidence.
How is comparing production cohorts different from running evals?
Evals check known cases against a fixed dataset, which is the right way to catch regressions before you ship. Cohort comparison checks whether real users hit fewer dead ends after the change, including cases nobody wrote a test for. Use both: offline first, then production.
How long should I wait before deciding a change worked?
Long enough to see a trend, not a point. For many agents that is one to two weeks of comparable traffic, depending on volume. A drop in week one that holds in week two is credible; a one-day dip is noise.
Do I need an A/B testing framework to compare agent versions?
No. A version tag on every event and a fixed success definition are enough for a before-and-after comparison. Splitting live traffic between versions removes more confounders, but it is an upgrade, not a prerequisite.