How to Know If Your AI Coding Assistant Is Helping Users Ship or Just Spinning
Here’s a scenario that happens every day on AI coding platforms. A user comes in with a bug. They describe it. The AI generates a confident, syntactically valid, totally plausible fix. The user copies it. They run it. It doesn’t work. Maybe it breaks something else. They come back, describe the new error. The AI generates another fix. Still broken. Six turns later the user closes the tab and your session log shows a completed session with high engagement.
Your metrics look great. The user got nothing.
This is the core analytics problem for teams building AI coding tools right now, and most platforms are completely blind to it. You’re measuring code generation. Your users are trying to ship software. Those are not the same thing, and the gap between them is where user trust goes to die.
After tracking millions of coding agent conversations at Agnost, the pattern is unmistakable. The platforms that retain developers are the ones who’ve figured out how to distinguish actual help from productive-looking noise. The ones who haven’t are watching their best users churn quietly while their engagement dashboards look fine.
^ your product analytics while users spend 45 minutes in an error loop your session metrics called “high engagement”
The “plausible but wrong” problem nobody talks about
AI coding assistants are genuinely good at one specific thing: generating code that looks right. Syntactically valid. Stylistically consistent. The kind of output that passes a quick skim. For experienced developers, this is a minor annoyance because they can spot the problem fairly quickly. For less experienced developers, the ones your vibe coding platform is probably targeting, its a trap.
They see the output. It looks like what they asked for. They copy it and run it. And then something breaks.
The session ended “successfully” by every conventional metric. Code was generated. The user accepted it. Tokens were consumed. But the user spent the next hour debugging AI-generated code that was confidently wrong. That’s negative value. The AI made their workflow worse, not better.
GitClear’s research on AI-generated code found that code churn (code that gets rewritten or deleted within two weeks of being written) has roughly doubled since AI coding tools became mainstream. That’s not a bug in their data. That’s the plausible-but-wrong problem showing up at the repo level.
For AI coding platform builders, this is the hardest quality problem you have. You can’t catch it with acceptance rate. You can’t catch it with session count. The only way to see it is through what happens to the conversation after the code is generated.
What you’re probably tracking (and why it’s not enough)
Let’s be honest about the metrics that dominate AI coding tool dashboards right now.
Code acceptance rate. Measures whether the user saw enough value in the AI’s suggestion to accept it in the moment. Tells you nothing about whether the code worked. Users, especially newer developers, accept plausible-looking output all the time and discover problems only when they run it. Acceptance is an input metric. Shipping is an output metric. These are not correlated the way you want them to be.
Session count and DAU. Measures usage. A user who spends 40 minutes in an error loop counts the same as a user who solved their problem in 5 minutes and shipped. You’re not measuring success, you’re measuring presence.
Tokens generated. This is a cost metric dressed up as a product metric. More tokens doesn’t mean better outcomes. In error loops, you’re often generating MORE tokens precisely because the AI keeps failing to fix the problem.
Time in session. Longer is not better if users are spending that time debugging code the AI got wrong. I’ve seen platforms celebrating “increased session depth” while simultaneously watching retention drop. The depth was all error loop. There’s no ambiguity about this in the data once you look at it at the conversation level.
None of these metrics are useless in isolation. But they all measure the AI generating code, not the developer shipping something. You need signals from the other side of that gap.
The 5 signals that actually tell you if the AI is helping users ship
These are the patterns we use at Agnost to distinguish real development velocity from spinning. Not all five are easy to implement from day one. Start with the first two if you have limited instrumentation bandwidth.
Signal 1: Error loop rate
If a user pastes an error message, gets a fix, and then pastes the same or closely related error message back into the conversation within the same session, the first fix didn’t work.
That sequence is your clearest signal of spinning. The AI generated output confidently. The user tried it. It failed. Now they’re back at square one but two turns deeper.
Track what percentage of your sessions contain this pattern. If you see a user describe an error, receive a code change, and then describe an error again within 3-4 turns, that’s an error loop. High error loop rate in a specific intent category (debugging, say, vs. code generation) tells you exactly where your AI is underperforming.
Signal 2: Clarification requests per task, over time
As a user works on a specific problem in a session, the number of clarifying questions they have to ask should go DOWN as the conversation progresses, because the AI is building a coherent understanding of their context.
If clarification requests are going UP mid-conversation, the AI is losing the thread. It’s not maintaining context. The user is getting more lost, not less. This is the subtle early signal that distinguishes a converging solution from a diverging one.
“Wait, which file did you mean?” “But I said it was a React component, not vanilla JS.” “I already told you the API endpoint changes, didn’t you see that?” These are conversation-level signals of context collapse, and they show up as increasing clarification density in sessions that are about to go sideways.
Signal 3: Task completion within expected turn depth
Simple tasks should resolve in 2-4 turns. Medium complexity tasks maybe 5-8. If a “change the button color” type request is taking 10 turns, something is broken. Either the AI doesn’t understand the codebase context, it’s hallucinating about the file structure, or it keeps generating fixes for the wrong component.
The key insight here: categorize your incoming tasks by complexity and track median turns-to-resolution per category. When you see a category with dramatically higher median turns than expected complexity would suggest, that’s a spinning signal for that category specifically. You’ve just identified a weak spot in your AI that’s costing you retention.
^ “high session depth” and “user stuck in an error loop,” simultaneously, in your dashboard
Signal 4: Session continuity after code generation
After the AI generates code and the user accepts it, does the user continue in the same session to implement and iterate? Or do they end the session shortly after accepting?
Session continuation after acceptance is a positive signal. It suggests the user got useful output and is moving forward. Session termination shortly after acceptance is ambiguous at best, and often negative. The user accepted optimistically, tried the code, found it didn’t work, and left.
The telling pattern: short sessions that end right after acceptance, followed by a new session with the same or similar problem. The gap between sessions where the user tried the code and discovered it failed is invisible to you unless you’re tracking same-problem recurrence.
Signal 5: Return on the same problem
This one takes a bit more infrastructure to track but its worth it. If a user opens a new session and describes a problem that’s substantively similar to a problem they addressed in a session within the last 48-72 hours, the previous session’s solution didn’t hold.
Return-on-same-problem is the most honest measure of whether the AI actually helped. Not “did the user accept the code.” Did the code actually work? Did they have to come back? The data is there in your session history, you just have to look for it.
We’ve seen products where 30%+ of sessions are users returning to a problem they apparently “resolved” in a previous session. Those aren’t high engagement numbers. Those are users whose AI keeps failing them in ways that look like success.
What spinning looks like in the conversation data
If you pull the conversation logs from your churned users and look at their last few sessions, you’ll almost always see one of these patterns.
The classic error loop: user describes an error, AI generates a fix, user pastes either the same error or a new error caused by the fix, AI generates another fix, repeat. Some of these sessions go 15+ turns with zero progress. The user kept trying because they’re persistent or because they thought maybe the next response would be the one that worked. Eventually they gave up.
Specificity regression: users who start a conversation describing their problem at a high level and have to get more and more specific because the AI keeps generating solutions to a slightly different problem than the one they have. “No, I meant the SUBMIT button, not the cancel button.” “No, the async version of this function.” “No, I’m using TypeScript not JavaScript.” Every clarification is evidence the AI didn’t maintain context.
Abandonment after acceptance: users accept code, session ends within a minute or two, they’re back with the same problem a few hours later. This is the silent failure mode. The session looked successful. The user found out privately, with no feedback loop to your platform, that it wasn’t. You lose the diagnostic data and then you lose the user.
Building a development velocity metric that actually means something
If you want a single number that proxies for “is the AI helping users ship,” here’s how to construct it.
Development velocity, at the session level, is roughly: tasks successfully completed per session, where “successfully completed” means the task was described, code was generated, there was no subsequent error loop, and the user moved on to a different topic or ended the session with a positive signal.
The formula isn’t perfect but its directionally honest. A high-velocity session is one where the user came in with problems and left having made progress. A low-velocity session is one where the user came in with a problem and left with the same problem, or worse.
Track this per intent category. Debugging tasks. Feature generation. Refactoring. Boilerplate. API integration. Your AI will be better at some of these than others, and the velocity metric will show you exactly which ones. This is the most actionable output you can generate from conversation analytics for your engineering and product teams.
High velocity categories: great, your AI is genuinely helping here. Low velocity categories: your model, your prompts, or your context handling is broken for this use case. Fix those first. Not the feature requests. Not the NPS comments. The low-velocity categories in your conversation data.
What this unlocks for your product decisions
Once you’re tracking development velocity and the five signals above, three things become immediately actionable.
Model and prompt prioritization. You know exactly which intent categories your AI is failing in. You stop guessing about where to focus your engineering attention and start making data-driven decisions about which prompts to improve or which fine-tuning data to collect.
User segmentation. Power users and “spinning” users can look identical in DAU and session count data. They look completely different in velocity data. Power users have high velocity sessions, low error loop rates, low clarification density. Spinning users have the opposite profile. These two cohorts need different things from your product, and you can’t serve them well until you know which is which.
Churn prediction. Users with consistently low velocity scores across their last 5 sessions are at high risk of churning. The data is there 2-3 weeks before the cancellation happens. You have a window to intervene, to show them better results, to route them to human support, to trigger a feature that helps with their specific problem category. But only if you’re looking at velocity, not session count.
The practical problem with building this yourself
Look, the instrumentation above is doable. Its not trivial but a good engineering team can get there. The harder part is that most teams building AI coding tools are already swamped with the actual AI infrastructure, the model integrations, the context management, the IDE plugins. Adding a conversation analytics pipeline on top of that, with the semantic similarity matching for error loop detection, the turn-level intent classification, the session continuity tracking, is real engineering work.
This is one of the specific problems we built Agnost for. Not generic product analytics bolted onto AI, but an analytics layer that understands the conversation structure of coding sessions natively. Error loop detection, clarification density tracking, velocity scoring by intent category, same-problem recurrence across sessions. The signals described in this post are the core of what Agnost tracks for AI coding platforms.
If you’re tired of your dashboard telling you engagement is up while your best users quietly leave, it might be worth 20 minutes to see what your conversation data actually shows.
^ every AI coding tool founder when they first see how much of their “successful sessions” were actually error loops
Where this is all going
The AI coding market is going to bifurcate. There are tools that generate code and tools that help developers ship. Right now most platforms are still in the first category, and they’re measuring themselves like they’re in the first category.
The ones who survive long-term will be the ones who close the measurement gap. Who understand that accepting a suggestion is not the same as using it to solve a problem. Whose product decisions are driven by development velocity and error loop rates, not tokens generated and sessions completed.
GitClear’s data on 41% higher code churn in AI-assisted codebases isn’t a problem for developers. It’s a problem for AI coding platforms. That churn is happening in your users’ repos after they used your tool. You can either measure it and fix it or you can keep tracking acceptance rate and wonder why retention is soft.
Wrapping it up
The gap between “the AI generated code” and “the user shipped something that works” is where your product either wins or loses. Standard metrics dont show you that gap. Conversation-level signals do.
Start with error loop rate. If you only add one metric from this entire post, make it that one. Pull the last 30 days of sessions, flag every one where a user described an error, got a response, and then described the same or a related error again within 3 turns. That number will tell you more about your AI quality problem than acceptance rate ever will.
Then build toward the full velocity picture. The teams who figure this out early are going to have a real structural advantage. Not because they have better models, but because they actually know what’s happening in their users’ sessions.
^ you, once you’ve got error loop rate, velocity scoring, and same-problem recurrence running and you actually know which 20% of your intent categories are responsible for 80% of your churn
If you’re building an AI coding tool and want to see what your conversation data actually looks like across these signals, Agnost tracks all of this natively. Error loop detection, clarification density, velocity by intent category, session continuity. Take a look at agnost.ai.
TL;DR: Code acceptance rate measures the AI generating output. Development velocity measures users shipping things that work. The gap between those two metrics is where your churn lives. Track error loop rate, clarification density, task completion depth, session continuity, and same-problem recurrence. These signals tell you which intent categories your AI is actually helping with and which ones are just spinning your users wheels.
Reading Time: ~9 min