Monitoring an AI agent means understanding what a user requested, how the application executed it, and what happened afterward. The instrumentation boundary changes with the framework. A persistent graph needs pause and resume context. A streaming assistant needs completion and cancellation handling. An optimized program needs the identity of the artifact that actually served the request.
This collection explains those differences and how to observe them with Agnost AI, an agent analytics, observability, and monitoring platform. Each guide includes an integration boundary, production failure patterns, and the limits of the evidence it records.
Choose your framework
| Framework | Start here | Main question the guide addresses |
|---|---|---|
| CrewAI | Tasks, delegation, and outcomes | Which part of the crew caused the result to go wrong? |
| LangGraph | State, interrupts, and retries | Did the workflow finish, fail, or pause intentionally? |
| LangChain | Retrieval, tools, and answers | Did the agent receive and use the right evidence? |
| OpenAI Agents SDK | Handoffs and tool execution | Did routing and tools move the user toward completion? |
| Agno | Agents, teams, and sessions | Can you connect team activity with the user’s session? |
| DSPy | Optimized programs in production | Which program artifact produced a prediction? |
| Pydantic AI | Validation and real correctness | Is the output merely valid, or actually correct? |
| Vercel AI SDK | Streaming and completion | What did generation complete, and what did the user receive? |
| Mastra | Agent and workflow traces | Where did execution diverge from the intended action? |
| VoltAgent | Conversations and tool results | Can you trace an action back to its request? |
| Photon Spectrum (Spectrum-TS) | Messaging and delivery | Was a reply generated, accepted for sending, or delivered? |
Understand the two integration levels
Conversation tracking records the application boundary you choose: usually a user request and a public response. Agnost’s Python package is agnost; its TypeScript conversation package is agnostai. This approach gives you deliberate control over identity and content. A wrapper does not automatically instrument the operations inside it.
Execution tracing captures supported agent, model, tool, and workflow operations through framework instrumentation and OpenTelemetry. It helps locate problems inside a request. Coverage depends on the framework, instrumentor, provider, and configuration. Confirm what a real exported run contains before building metrics around a field.
You can use both levels. Choose one canonical record for request counts so a public interaction, root span, and several child spans do not become several apparent user requests.
Define success at the right boundary
A returned function, a validated object, and a completed user task are different signals. A booking assistant may return normally even when no inventory is available. A document extractor may produce valid JSON with the wrong total. A messaging assistant may generate a reply that the channel never delivers.
Use execution status for technical reliability. Use authoritative application evidence for completed actions. Use reviewed or explicit feedback for outcomes that cannot be established automatically. Leave unknown and pending states visible.
This also changes how you investigate latency. A graph waiting for approval is not spending that entire interval generating tokens. Parallel tools do not make user latency equal the sum of their durations. A fast first token does not guarantee a fast complete answer.
Make one request explainable before expanding collection
For your chosen framework, inspect a representative interaction from input to terminal state. Confirm identity, timestamps, public output, and the relevant tool or workflow evidence. Then inspect a known failed or interrupted interaction. If the recording only works on the happy path, aggregate reliability will be misleading.
In Agnost, start with conversation review, then inspect the underlying events. Configured intents help organize what users wanted. Intent classification does not itself establish completion.
The goal is an explainable product interaction: who needed what, which execution served them, and what evidence shows the result. Choose the framework guide above that matches the boundary your application actually runs.