How do I find repeated failed attempts in chatbot conversations?

How to find repeated failed attempts

1. Group turns by conversation and by user. Send every turn with one stable conversation ID and a stable user ID. Same-session loops need the first; cross-session loops need the second. In Agnost AI those are conversation_id and user_id in the Python SDK, or agnost.session_id on an OpenTelemetry span.
2. Embed each user message and compare it with earlier ones. For every user turn, compute cosine similarity against each previous user turn in the conversation. Flag a turn when similarity to a prior turn exceeds your threshold (start around 0.85) and the agent replied in between. That is a repetition event.
3. Separate repetition from clarification. Discard pairs where the later message adds new information. A rephrasing restates the same request; a clarification narrows it.
4. Check across sessions. Bucket each user’s messages by session and run the same comparison across session boundaries, or count how often a user hits the same intent. Three or more sessions on one intent without a resolution is a cross-session repeat.
5. Flag agent-side retries too. The same tool called again with the same arguments after a failure is the agent’s version of a failed attempt. Look for repeated failing calls per conversation in your tool-call data.
6. Group flagged conversations by intent and by the agent turn before the repeat. If one response type or one tool precedes most repeats, that is the bug. Read ten examples before changing anything.
7. Fix by failure type and compare the repeat rate before and after. The fix table below maps each pattern to where the fix usually lives.

Repetition versus clarification

Clarification is healthy. “Can you help me write a cover letter?” followed by “make it more casual and about 150 words shorter” adds information and moves the conversation forward.

Repetition is different. “How do I cancel my subscription?” followed by “I want to stop my account and not be billed again” says the same thing in different words. The user is not building on the agent’s answer; they are trying to get through to it.

In embedding terms, clarifications usually score low to moderate similarity against the original message, and rephrasings score high. Pick a starting threshold around 0.85, then calibrate:

• For products where users phrase things formally and similarly (legal or compliance tools), raise it to 0.88 or 0.9 to avoid false positives.
• For products with casual, varied language, 0.82 may catch more real repeats.
• Spot-check a sample of flagged conversations before treating the metric as reliable.

There are two kinds worth tracking separately:

• Immediate repetition happens inside one session: the user asks at turn 3, the agent misses, the user rephrases at turn 5. The causal chain is visible in one transcript.
• Distributed repetition spans sessions: the user asks on Tuesday, gets an incomplete answer, and asks again on Thursday in different words. It only shows up at the user level, and when it does, it usually means the agent has a standing failure on one intent.

How a loop unfolds

The user asks something. The agent gives a coherent answer that misses the point. The user rephrases. The agent gives a different answer, still missing, often with more confident language. The user tries once more, gets something that sounds comprehensive but is built on the original miss, and leaves.

Counted by turns, that was an eight-turn “engaged” session. In reality it was a user who tried four ways to get one thing done and failed. Loops tend to cluster in a few places:

• Support bots on edge cases. The agent handles common requests well and struggles outside them. Users with an edge case rephrase several times, then escalate if they can or leave if they cannot.
• Coding assistants on niche problems. Obscure libraries, specific architecture constraints, or poorly documented APIs produce answers that address the question without solving it.
• Companions and tutors that lose context. When the agent acknowledges personal context without retaining it, users revisit the same ground repeatedly. Memory failures and repetition are often the same problem seen from two angles; see agent memory failures in production conversations.

Three patterns and where to fix them

Pattern	What it looks like	Usual root cause	Where the fix lives
Triple rephrase	Same intent, three phrasings, one session; often ends in abandonment or escalation	The agent is misreading the request	System prompt, intent handling, retrieval for that intent
Weekly groundhog day	Same intent across several sessions, never resolved	A standing failure on one intent	Targeted prompt or tool fix, validated on that intent’s repeat rate
Context reset	User re-explains their situation or preferences again and again	Context dropped in long conversations or across sessions	Context window handling and persistent user memory

If repeats cluster around specific intents, which they usually do, the fix is targeted: reprompt or add the missing tool or knowledge for those intents, then check whether that intent’s repeat rate moves. If repeats look like context resets, the fix is architectural. If the agent itself is retrying a failing tool, look at retry limits and duplicate-call handling; see how to detect and stop duplicate tool calls.

Why standard metrics miss it

Turn count does not say why turns happened. Satisfaction surveys reach a small, self-selected share of users, skewed toward strong reactions rather than the quiet user who gave up after three rephrases. Latency and token cost say nothing about whether the request was understood. Manual review of the worst sessions catches loops, but it stops scaling at a few dozen conversations a day.

Repetition is a strong in-conversation failure signal because users rarely rephrase after a good answer. It is also one input to a broader score; see how to measure user frustration in chatbot conversations.

Tools that do this

• Agnost AI. Keeps each conversation as one timeline of turns, tool calls, and errors, groups user-level activity by user_id, and lets you track loop patterns (“user repeats a request after an unhelpful answer”) as intents with linked evidence and alerts. Tool-call analytics show success rate and error distribution per tool for agent-side retries. There is no documented built-in similarity score, so you define the pattern. Free up to 1,000 events a month.
• Arize Phoenix. A pre-built User Friction evaluator that detects corrections and retries after an unsuccessful response at span, trace, or session level. Free to self-host under the Elastic License 2.0; you run the evaluator and log results.
• PostHog. An LLM-judge “User correction” template, runnable per trace or per session, next to session replay and product analytics. Needs your own LLM provider key, and full prompt and response text is deleted after 30 days.
• Dimension Labs (dashbot.io now redirects there). An “AI Failure Mode” field that flags loops and repeated answers, alongside containment and escalation analysis. Built for platform bots such as Ada or Dialogflow CX; no trace or tool-call ingestion, from $299/month.

For more options, see the best conversation analytics tools for AI agents.

Start with one export

You can run the same-session check offline today: export a week of conversations, embed user turns, flag pairs above 0.85 with an agent reply in between, and group the flags by intent. Then read the worst ten. For the wider set of behaviors that mean a user is stuck, see where users get stuck with an AI agent.

To see repeated questions and tool failures on one timeline for your own agent, start with Conversations in the Agnost AI docs.

Frequently asked questions

How do I detect when a user rephrases the same question to a chatbot?

Embed every user message and compare each one with the earlier user messages in the same conversation. Flag a turn when its similarity to an earlier turn is above your threshold and an agent reply came in between. Read a sample of flagged pairs to confirm they are rephrasings, not follow-ups.

What similarity threshold should I use to detect repeated messages?

Start around 0.85 cosine similarity and calibrate on your own transcripts. Products where users phrase things formally may need 0.88 to 0.9 to avoid false positives, while products with casual, varied language may catch more real repeats near 0.82. The right number depends on your embedding model and your users.

Is repetition always a sign the chatbot failed?

No. Clarification adds new information, such as a length limit or a different tone, and moves the conversation forward. Repetition restates the same request in different words because the first answer missed. Only the second is a failed attempt.

How do I find users who ask the same thing across sessions?

Group conversations by a stable user ID, then run the same similarity check across session boundaries or count how often each user hits the same intent. A user who raises the same intent in three or more sessions without a resolution is a cross-session repeat, which single-session views never show.