---
title: "How do I see what users do in my AI agent?"
description: "To see what users do in your AI agent, log whole conversations instead of page events: every message, reply, and tool call under one conversation ID and user ID. Then read individual conversations and group them by what users asked for. Agnost AI does this with conversation timelines and intents; PostHog, Langfuse, and Amplitude offer similar views."
author: "Agnost AI Team"
datePublished: "2026-02-25T00:00:00.000Z"
dateModified: "2026-10-29T00:00:00.000Z"
canonical: "https://agnost.ai/blog/tracking-conversations-like-pageviews/"
tags: ["Conversational AI Analytics","AI Product Metrics","Conversation Analytics","AI Chatbot Tracking","Product Analytics","User Behavior"]
---

# How do I see what users do in my AI agent?

To see what users do in your AI agent, log whole conversations instead of page events: every message, reply, and tool call under one conversation ID and user ID. Then read individual conversations and group them by what users asked for. Agnost AI does this with conversation timelines and intents; PostHog, Langfuse, and Amplitude offer similar views.

## How to see what users do in your AI agent

1. **Log whole conversations, not only events.** Send every user message and agent reply with one conversation ID reused across turns. A `message_sent` event tells you a message existed; the text tells you what the user wanted. In Agnost AI that is `conversation_id` in the SDK or `agnost.session_id` on an [OpenTelemetry span](https://docs.agnost.ai/otel).
2. **Attach a stable user ID to every turn.** This lets you follow one person across conversations and compare groups of users.
3. **Put tool calls and errors in the same timeline.** When a user gives up, the cause is often a failed lookup or a slow tool call two turns earlier, which you can only see if it sits next to the messages.
4. **Read individual conversations first.** Open a dozen from yesterday, start to finish. This is the AI-agent equivalent of watching session replays, and it will show you patterns no chart has a column for yet.
5. **Group conversations by what users wanted.** Name the goals in plain language ("reset a password", "compare plans") and classify every conversation against them, so you can count demand and see which goals fail most.
6. **Flag where conversations go wrong.** Rephrasing loops, mid-task abandonment, requests for a human, and rules the agent broke. Measure the share of conversations affected, not raw counts.
7. **Add metadata you will want to slice by.** Plan, surface, agent version, and experiment arm turn "users are unhappy" into "users on the new prompt are unhappy."

## What a pageview-style event log hides

Most teams instrument an AI product the way they would a web app:

```
conversation_started   { user_id: 123, timestamp: 10:04:12 }
message_sent           { user_id: 123, turn: 1 }
message_sent           { user_id: 123, turn: 2 }
message_sent           { user_id: 123, turn: 3 }
conversation_ended     { user_id: 123, duration: 4m32s }
```

Three very different conversations produce exactly that log:

- **Story A.** The user asked a coding question, got a correct answer, asked a follow-up, said thanks, and left satisfied.
- **Story B.** The user got a wrong answer, rephrased, got another wrong answer, rephrased again, and closed the tab mid-sentence.
- **Story C.** The agent kept confidently answering a different question. The user tried three phrasings, gave up, found the answer elsewhere, and did not come back.

Story A is the product working. Story B is a quality problem. Story C is churn in progress. The dashboard shows three healthy four-minute sessions.

The pageview model assumes the content sits still and the user moves through it, so more activity means more value. A conversation breaks that assumption. A user stuck in a loop sends many messages and stays a long time, which every engagement metric reads as a power user. A user who got a good answer in one turn and left looks casual.

## Event metrics and what to track instead

| Event metric | What it actually tells you | Conversation-level replacement |
|---|---|---|
| Session count | How many conversations started | Conversations per user goal (intent) |
| Session length | Time spent, including time spent stuck | Share of conversations with a rephrasing loop |
| Turn count | How many messages were exchanged | Turns before the user's goal was met, or before they left |
| DAU/MAU | How often users open the product | Whether returning users come back with new goals or the same failed one |
| Bounce rate | Whether they left quickly | Whether a quick exit followed a good answer or a miss |

The right-hand column needs a tool that understands conversation structure: what each turn was, what the user was trying to do, and how it ended. None of it requires new data. It requires reading the text you already have.

## A conversation is a task attempt

The useful reframe: a conversation is not a pageview or even a session. It is a task attempt. The user arrived with a goal, tried to get it done through the agent, and either did or didn't.

That is how support teams are measured. Nobody counts how many tickets an agent opened or how long each call lasted; they count how many problems got solved. An AI agent deserves the same treatment, and the analytics should be organized around goals and outcomes rather than activity.

Two consequences follow. A short conversation can be a great outcome (fast resolution) or a terrible one (instant give-up). A long one can be deep engagement or a frustration spiral. Only the content tells you which, so the content has to be in your analytics.

Getting to an outcome number takes one more step: you define what success means for each goal. Agnost AI does not compute a built-in task-success rate. You express success as a rule, mark failed interactions with the `success` field, and track tool-call success rates; see [how to measure agent task success without inflating the number](/blog/measure-agent-task-success-rate/).

## Tools that do this

- **Agnost AI.** Groups each conversation into one timeline of turns, tool calls, errors, and audio (the "User Stories" view), classifies conversations against [intents](https://docs.agnost.ai/using-intents) you write in plain language and suggests new ones from observed traffic, and checks rules you define. Free up to 1,000 events a month.
- **PostHog.** AI Observability groups traces by `$ai_session_id`, clusters similar traces automatically, and links each one to the user's regular session replay and product events. The best fit if you already run PostHog. Full prompt and response text is deleted after 30 days, and clustering needs at least 1,000 traces a week.
- **Langfuse.** Open-source (MIT core) sessions with a replay of the whole interaction and a per-user view of traces and feedback. Intent labels are not built in: Langfuse's own guide suggests running your own classification pipeline and writing labels back as scores.
- **Amplitude Agent Analytics.** Agent sessions with topic clustering and a built-in User Intent signal, inside the same charts as your product events. Conversation text is kept 14 days on Free and Plus.
- **LangSmith.** Threads group runs into conversations, and Insights clusters production traces into usage patterns and failure modes. Insights is on Plus and Enterprise and samples up to 1,000 traces per report.

For a full comparison, see [the best conversation analytics tools for AI agents](/blog/best-conversation-analytics-tools-ai-agents/). For the reading-transcripts side of this, see [session replay for AI agent conversations](/blog/session-replay-for-ai-agent-conversations/), and for grouping by goal, [how to group chatbot conversations by user intent](/blog/auto-generated-intents-ai-product-metrics/).

To open your first conversation timeline, follow the [conversations guide](https://docs.agnost.ai/using-conversations) after the [quickstart](https://docs.agnost.ai/quickstart).

## Frequently asked questions

### How do I track what users do in my AI chatbot?

Log every user message, agent reply, and tool call with one stable conversation ID and user ID, then use a tool that shows each conversation as a single timeline. Once you can read individual conversations, group them by what users were asking for so you can count patterns instead of reading transcripts one by one.

### Can I use my existing product analytics to track an AI agent?

Partly. Click and page analytics record that a chat happened, how long it lasted, and how many messages were sent, but not what the user wanted or whether they got it. Amplitude and PostHog now have agent features that bring conversations into product analytics, and Mixpanel has one in beta.

### Is there session replay for AI agent conversations?

Yes, in the form of conversation timelines. Agnost AI, Langfuse, LangSmith, and PostHog group turns and tool calls by a session or conversation ID so you can read the whole interaction in order. PostHog also links the trace to the user's regular browser session replay.

### What should I log from my AI agent to understand users?

At minimum: the user message, the agent reply, each tool call with its success state and latency, a conversation ID reused across turns, and a stable user ID. Add metadata such as plan, surface, and agent version so you can compare groups later.