Why Your Agent’s Success Rate Tells You Nothing About Agent Experience
Your agent completes 87% of tasks. Your eval suite is green. Your team is celebrating.
And your users are quietly losing faith in the thing you built.
This is the trap. Task completion rate feels like the right metric because it’s clean, it’s measurable, and it maps directly to what agents are supposed to do: complete tasks. Every team building an AI agent starts there, and its not a bad starting point. The problem is when it becomes the ending point too. When 87% becomes the number you optimize for, present in board decks, and use to decide if your agent is working.
Because an agent can complete a task and still deliver an experience so painful, inefficient, or trust-eroding that the user would rather go back to doing it manually. You just cant see that in your completion rate.
^ your eval dashboard, while your users quietly decide they’d rather just do the task themselves
Why every team starts with task completion rate (and why thats fine)
Look, there’s a reason success rate is the first thing you instrument. It’s the most legible signal you have for a brand new agent.
Before you know what users will actually ask, before you have enough volume to detect patterns, before you’ve built trust in any other metric, completion rate is the baseline sanity check. Is the agent doing the thing? Or is it catastrophically failing half the time and refusing tasks it should be able to handle?
That question matters. If your completion rate is 40%, you have bigger problems than agent experience. Fix the agent first.
But the moment you get north of 70-75%, completion rate starts losing its edge as a signal. Your agent is “working” in the sense that it’s finishing tasks. What you lose visibility into, completely, is whether completing those tasks is actually making your users more capable.
And that’s the whole game with agents. Not just task completion. Capability amplification.
The 4 ways an agent can “succeed” while destroying the experience
This is where it gets uncomfortable. Because these aren’t edge cases. We see all four of these patterns regularly in agent conversation data at Agnost AI.
1. Completed the task the wrong way
The agent finished. The output is technically correct. But it took 14 tool calls, looped twice on a subtask, re-fetched data it already had, and burned 80,000 tokens to do something that should have taken 8,000.
The user got their answer. But they waited twice as long as they should have, and the cost per task is quietly killing your unit economics.
Path efficiency doesn’t show up in completion rate at all. A lean 3-step completion and a sprawling 14-step meander through the same problem look identical in your success metrics. But they’re completely different agent experiences, and your users feel the difference even if they can’t articulate it.
2. Completed the task but eroded user trust
This one is the slow poison.
The agent gave a confident answer. The answer was mostly right. But it got one detail wrong, or hallucinated a source, or made an assumption the user didn’t ask it to make. The user caught it, maybe. Or their colleague caught it. Or their client caught it.
Next time they use the agent, they check every output more carefully. The time after that, they double-check again. Before long, every agent output becomes a first draft they review line by line rather than a result they trust and act on.
The completion rate is still 87%. But the user’s delegation behavior has completely changed. They used to hand off tasks. Now they supervise every one. The agent hasn’t gotten worse. The trust has.
And you have no visibility into this at all unless you’re looking at delegation depth over time.
^ the moment you realize your “high completion rate” users are manually reviewing 100% of the agent’s outputs
3. Completed the easy version of the task, missed the actual intent
This is the sneaky one.
User asks the agent to “prepare a summary of the Q3 sales data.” The agent produces a summary. Technically done. Completion rate +1.
But the user’s actual intent was to understand which regions underperformed and why, because they have a stakeholder meeting in two hours. The agent gave them a descriptive summary when they needed an analytical one. The task completed. The intent missed entirely.
There’s a version of this in every agent category. The research agent that summarizes without synthesizing. The coding agent that generates code that compiles but doesn’t match the architectural patterns the codebase uses. The scheduling agent that finds a meeting time but doesn’t account for travel time before the next appointment.
Successful task. Failed intent. Your success rate tracked the former and is blind to the latter.
4. Technically succeeded but the user still had to do significant rework
Here’s the real world test case.
An AI coding assistant that completes 85% of tasks but generates code that users rewrite 60% of the time. Is that a good agent experience?
GitClear’s research on AI-assisted development found that code churn (code written and then rewritten or deleted within two weeks) has roughly doubled since AI coding tools went mainstream. That’s the rework problem showing up at the repo level. And before anyone argues that rewriting is natural, it wasn’t this prevalent before agents started generating the first draft.
When an agent produces output that requires substantial rework, the completion rate says 1. The user’s actual experience says: “I got a starting point, not a result.” Those are different things. The latter shouldn’t count as a full success, and measuring it as one is how you end up making product decisions based on data that’s lying to you.
What success rate actually measures (and what it doesn’t)
Here’s the honest framing. Task completion rate measures a binary: did the agent produce an output that meets the stated criteria?
It doesn’t measure:
- How much time the agent took relative to what the task warranted
- Whether the output was used as-is or heavily modified
- Whether the user is more or less likely to delegate similar tasks next time
- Whether the approach taken was efficient or wasteful
- Whether the user’s actual underlying goal was served
- Whether trust in the agent is growing or eroding over time
In other words, it measures the mechanical completion of the task and nothing about the felt experience of using the agent. For a product that’s supposed to make users more capable, that’s a pretty significant gap.
The teams that figure this out early get a structural advantage. Not because they have a better model. Because they’re actually measuring the right things.
The eval trap: optimizing for the wrong thing
The pattern that keeps showing up in conversations with teams building agents: they run evals that are essentially automated completion rate checks. Does the agent complete this class of task? Does it produce output that meets these criteria? Green. Ship it.
Meanwhile, in production, the experience is degrading.
Not in ways that show up as failures. In ways that show up as increased verification behavior, decreased delegation depth, higher rework rates, and eventually, quietly, users choosing to handle tasks themselves again because “the agent is more trouble than it’s worth.”
You built toward 90% completion rate in evals. You got it. And now you’re watching DAU flatten because the experience you optimized for isn’t the experience that drives retention.
The trap is that evals are measuring the thing that’s easy to measure, not the thing that matters. And the thing that matters, the felt experience of using the agent, whether users are becoming more capable or less, whether trust is building or eroding, those dont fit neatly into a benchmark suite.
What to measure instead (without throwing out success rate)
To be clear: don’t stop tracking completion rate. You need it as a baseline. The argument isn’t that success rate is useless, its that success rate alone is a lie of omission.
Here’s what to track alongside it.
Path efficiency. For any given task type, what’s the median number of steps, tool calls, and tokens your agent uses to reach completion? Track this as a distribution, not just a mean. When your 85th percentile looks dramatically worse than your median, you have an agent that succeeds reliably but sometimes goes completely off the rails. Users in that long tail are having a very different experience than your aggregate metrics suggest.
Trust retention signals. This is harder to instrument but not impossible. Look for behavioral patterns that indicate a user is verifying rather than trusting: follow-up conversations that re-examine completed tasks, explicit “can you double-check” type requests that appear after a previous task, declining delegation scope over time. Users who trusted your agent with complex tasks last month but are now only delegating simpler ones are a churn signal hiding inside a success rate.
Delegation depth over time. The clearest measure of whether your agent is actually working: are users giving it harder tasks over time, or easier ones? An agent that’s genuinely building user capability should see delegation depth increase as users develop confidence. An agent that’s eroding trust will show the opposite, users gradually retreating to only the safest, most verifiable tasks. This is the metric that tells you if your agent is actually changing how people work or just completing the tasks they were already comfortable delegating.
Across agent conversations we track at Agnost AI, the strongest leading indicator of agent churn isn’t a spike in failure rate. It’s a sustained decline in the complexity of tasks users are willing to delegate. The agent kept “succeeding” the whole time. But the user had already decided it wasn’t worth trusting with anything important.
The practical problem with building this visibility yourself
Here’s the honest part. Instrumenting path efficiency, trust signals, and delegation depth is not trivial. Its real work on top of the agent infrastructure work you’re already doing. And the teams that need this visibility most, early-stage teams figuring out product-market fit, are usually the ones with the least bandwidth to build custom analytics pipelines.
This is exactly the gap we built Agnost AI to close. Not just logging agent runs, but tracking the experience layer that completion rate can’t see. Path efficiency distributions by task type, trust erosion signals at the user level, delegation depth trends over time. The metrics that tell you whether your agent is actually making users more capable, not just finishing tasks.
If you’re hitting the eval wall, where your benchmarks look great but something feels off in how users are actually engaging, its worth seeing what your conversation data shows beyond the success rate.
Where this is heading
The agent market is about to bifurcate the same way the AI coding tool market is bifurcating.
There will be agents with great completion rates and agents with great agent experiences. For the first year or two, those look similar from the outside. In year three, one of them has deeply loyal users who keep increasing their delegation depth, and the other has a retention problem nobody can explain because the success rate was always fine.
The teams that build the measurement layer now, who instrument beyond success rate while they still have time to act on what they find, are the ones who will understand their agent products deeply enough to actually fix them.
Your 87% completion rate is not the story. It’s the opening line. The story is in what happens after the task completes.
Wrapping it up
Task completion rate is not a bad metric. It’s an incomplete one. And the incompleteness is specifically in the direction that matters most: whether the user experience of your agent is building the kind of trust that keeps people coming back and delegating more.
An agent that completes tasks while eroding trust, missing intent, or generating output that requires constant rework is not a good agent experience. It’s a treadmill dressed up as progress. And the longer you optimize for the completion rate without tracking what sits behind it, the longer you’re flying blind on the thing that actually drives retention.
Path efficiency. Trust retention. Delegation depth. Track those alongside your success rate, and you’ll actually know whether your agent is working.
^ you, once you can finally see what’s happening in your agent sessions beyond the completion rate
Stop guessing about whether your agent is building user trust or quietly eroding it. Agnost AI tracks path efficiency, delegation depth, and the conversation-level signals that success rate can’t show you. See what your agent data actually looks like at agnost.ai.
TL;DR: An 87% task completion rate means your agent is finishing tasks. It says nothing about whether it’s doing it efficiently, building user trust, hitting the actual intent, or producing output worth using. Track path efficiency, trust retention signals, and delegation depth alongside success rate, or you’re optimizing for the thing thats easiest to measure instead of the thing that drives retention.
Reading Time: ~8 min