---
title: "How to Diagnose AI Agent Latency Beyond Time to First Token"
description: "Measure agent latency from request to useful outcome. Separate model time, tools, queueing, approval waits, and the critical path before optimizing."
author: "Agnost AI Team"
datePublished: "2026-10-17T00:00:00.000Z"
dateModified: "2026-10-29T00:00:00.000Z"
canonical: "https://agnost.ai/blog/ai-agent-latency-critical-path/"
tags: ["Agent Monitoring","Performance","Tool Calls"]
---

# How to Diagnose AI Agent Latency Beyond Time to First Token

Measure agent latency from request to useful outcome. Separate model time, tools, queueing, approval waits, and the critical path before optimizing.

**Measure agent latency from the user's request to the useful outcome, then inspect the work on that critical path.** Time to first token measures when text begins. It does not tell you when a booking is confirmed, a report is ready, or the agent has enough evidence to answer correctly.

A quick “Let me check” can make an agent look responsive while the actual task remains slow.

## Use timestamps with explicit meanings

Record when the request arrives, the first meaningful output appears, each tool starts and ends, the final answer finishes, and the required business outcome becomes available. Use a monotonic clock for local durations and correlated trace timestamps for distributed investigation.

| Measurement | What it answers |
| --- | --- |
| Request to first token | How soon does output start? |
| Request to useful answer | When can the user act on the response? |
| Request to confirmed outcome | When did the external task finish? |
| Active compute/tool time | How much execution work occurred? |
| Approval or queue wait | How much time was spent waiting? |

For streaming interfaces, define “useful” in a way the application can measure. It may be the first complete field or the final response, not a universal token boundary.

## Do not sum overlapping spans

Consider an illustrative run with two parallel reads: one takes 800 milliseconds and the other 1,000 milliseconds. A final generation takes 400 milliseconds after both finish. Ignoring other overhead, the path is 1,400 milliseconds, not 2,200.

Now suppose an unnecessary third lookup adds 600 milliseconds after the parallel reads. Removing it may shorten the critical path by 600 milliseconds. Removing a 600-millisecond operation that was already hidden behind longer parallel work may save compute without changing response latency.

[OpenTelemetry traces](https://opentelemetry.io/docs/concepts/signals/traces/) represent operations as spans with timing and relationships. Use those relationships to reconstruct dependencies. A chart of average tool durations alone cannot identify which operation blocked the user.

## Break down slow runs before changing the model

Inspect queueing, retrieval, tool execution, repeated work, generation, and downstream confirmation. Compare slow and typical runs for the same task type. A complex research request and a single account lookup should not share an unexplained latency target.

| Observation | Investigation |
| --- | --- |
| Slow before the first tool starts | Queue, authorization, context preparation |
| Many identical reads | Lost observations, duplicate calls, cache policy |
| One long external call | Provider latency, timeout behavior, payload size |
| Long pause awaiting approval | Human workflow rather than model execution |
| Fast answer, late action confirmation | Asynchronous outcome handling |

Change one cause at a time and check quality. Skipping retrieval may make answers faster while increasing unsupported claims. Parallel execution may reduce latency but create unnecessary calls or violate dependencies.

## Compare distributions carefully

Report p50 and p95 or another percentile appropriate to your volume, alongside the number of runs and task mix. Do not average instance-level p95 values to obtain a fleet-wide p95. [Prometheus documents the aggregation differences between histograms and summaries](https://prometheus.io/docs/practices/histograms/); choose instrumentation that supports the comparisons you need.

Include cancellations and timeouts explicitly. Dropping unfinished runs can make the slowest behavior disappear from the dashboard. For workflows awaiting people, report active execution and total elapsed time separately.

## Verify that the faster version still completes the task

Use a fixed regression set for correctness and a comparable workload for latency. Keep model configuration, tool environment, and cache state visible in the report. If the cache is warm in one run and cold in another, explain that difference.

Agnost's [tool analytics](https://docs.agnost.ai/using-tool-calls) can help identify slow or repeated operations in their conversation context. The [Vercel AI SDK monitoring guide](/blog/vercel-ai-sdk-monitoring-observability-guide/) covers useful streaming boundaries. For cost analysis alongside speed, continue with [cost per completed task](/blog/ai-agent-cost-per-completed-task/).