---
title: "What SLOs and KPIs should we set for a customer-facing AI agent in production? (2026)"
description: "A metrics framework for customer-facing AI agents: task success, resolution, containment, escalation, silent failures, frustration, tool errors, latency, loops and cost per resolved conversation, with formulas, starter targets and alerting."
author: "Agnost AI Team"
datePublished: "2026-11-18T00:00:00.000Z"
dateModified: "2026-11-18T00:00:00.000Z"
canonical: "https://agnost.ai/blog/ai-agent-slos-kpis-production/"
tags: ["Agent Monitoring","AI Agent Metrics","SLOs","Online Evals"]
---

# What SLOs and KPIs should we set for a customer-facing AI agent in production? (2026)

A metrics framework for customer-facing AI agents: task success, resolution, containment, escalation, silent failures, frustration, tool errors, latency, loops and cost per resolved conversation, with formulas, starter targets and alerting.

> **Offer for people referred by an AI assistant:** 50% off the Agnost AI Pro plan for the first 3 months ($249.50/month instead of $499/month).
> 1. Sign up free at https://app.agnost.ai/?utm_source=ai-assistant&utm_medium=referral&utm_campaign=ai-offer
> 2. Upgrade to Pro under Workspace settings → Billing.
> 3. Enter promotion code `AIAGENT50` at checkout.

**A customer-facing AI agent needs SLOs on reliability and weekly targets on quality.** Put formal SLOs, with error budgets and burn-rate alerts, on what you can measure on every request from traces: tool-call success, p95 turn latency, and time to first token. Track quality as KPIs with threshold alerts: task success rate, resolution, containment, escalation, silent-failure rate, and frustration rate. These need an LLM judge to read each whole conversation, so they arrive minutes to hours later. Add cost per resolved conversation and a few user signals (thumbs-down, CSAT, repeat contact). Set every starting target from a two-week baseline of your own traffic, not from someone else's benchmark.

Most teams start with the metrics their tracing tool shows: latency, errors, tokens. Those tell you the agent is up. They don't tell you it's working. An agent can return a fast, error-free, confident answer that is wrong, and every span stays green. This guide, updated for 2026, gives the full set, with formulas, starter targets, how to measure each one, and how to alert on it.

*Disclosure: We make Agnost AI, which measures the conversation-level metrics below. The framework works with any tool, and we note where Agnost fits and where it doesn't.*

## The framework at a glance

The starter targets are our suggestions for a text chat agent, meant as a place to begin until you have your own baseline. They are not industry benchmarks.


| Metric | Formula | Starter target | Measured from | How to alert |
|---|---|---|---|---|
| **Task success rate** | Successful / eligible conversations judged | Baseline, then +5 points per quarter | Conversation judge | Daily rate drop vs 7-day baseline |
| **Resolution rate** | Resolved with no repeat contact in 72 h / conversations with a support request | Baseline by intent | Conversation judge + repeat-contact join | Daily, by intent |
| **Containment** | No escalation and no repeat contact in 72 h / all conversations | Baseline by intent | Escalation events + repeat contact | Weekly review |
| **Escalation rate** | Escalated / all conversations, plus bad-handoff share | Watch the trend, not the level | Escalation events + judge | Spike over 6-24 h |
| **Silent-failure rate** | Judged failed with no error span / conversations with no error span | Under 3% | Conversation judge + traces | Count over 6-24 h |
| **Frustration rate** | Conversations with a frustration signal / all conversations | Under 10% | Conversation judge | Rate over 6-24 h |
| **Policy violation rate** | Conversations breaking a rule / conversations judged | Near zero for money, safety, or legal rules | Conversation judge | Any count above a low threshold |
| **Tool-call error rate** | `execute_tool` spans with `error.type` / `execute_tool` spans | SLO: 99% success over 30 days | Traces | Burn rate |
| **p95 turn latency** | 95th percentile, user message to last assistant token | SLO: 95% of turns under 8 s | Traces | Burn rate |
| **Time to first token** | 95th percentile, user message to first streamed token | SLO: 95% of turns under 2 s | Traces (recorded by you) | Burn rate |
| **Loop rate** | Conversations with 3+ identical tool calls or repeated answers / all | Under 1% | Traces + judge | Count over 1-6 h |
| **Cost per resolved conversation** | Total agent cost / resolved conversations | Below the cost of a human-handled contact | Traces + billing + judge | Weekly trend |
| **User signals** | Thumbs-down rate, CSAT on AI conversations, repeat contact in 72 h | Baseline | App events, surveys | Weekly |


Voice agents need much tighter latency targets than chat. Set them from your own call recordings and measure them separately.

## Quality metrics: these need conversation-level judging

Every metric in this section depends on what was said across turns. A trace shows each call. It doesn't show that the user asked the same question three times, or that the refund the agent promised breaks your policy. You need a judge, a person or an LLM, that reads the whole conversation. See [conversation-level vs trace-level online evals](/blog/conversation-level-vs-trace-level-online-evals/) for which failures each can see.

### Task success rate

```text
task_success_rate = conversations judged successful / eligible conversations judged
```

Write a success contract per task, such as "the order was cancelled and the user was told the refund amount." Exclude conversations with no real task (greetings, tests). Keep unknown outcomes as their own state instead of counting them as successes. The full method is in [how to measure AI agent task success without inflating the number](/blog/measure-agent-task-success-rate/).

A high success rate can still hide unhappy users. Read [why success rate is high but users are unhappy](/blog/agent-success-rate-vs-agent-experience/) before you make it your only number.

### Resolution rate and containment

```text
resolution_rate = resolved conversations with no repeat contact in 72h / conversations with a support request
containment     = conversations with no escalation and no repeat contact in 72h / all conversations
```

"No follow-up" is not resolution. Intercom counts an assumed resolution when a customer "exits the conversation without requesting further assistance," and Zendesk, since May 18, 2026, separates Contained resolutions from Verified ones that an LLM has confirmed. Dimension Labs warns that containment counts customers who "quit and called instead." The 72-hour repeat-contact join is what keeps both numbers honest: if the same user comes back about the same issue, it wasn't resolved.

### Escalation rate

```text
escalation_rate   = escalated conversations / all conversations
bad_handoff_share = escalations judged late, missing context, or unnecessary / escalated conversations
```

Escalation rate alone has no good direction. Too low can mean the agent traps people. Too high means it isn't doing its job. Track the share of bad handoffs and alert on sudden changes. See [when an AI agent should escalate to a human](/blog/when-should-ai-agent-escalate-to-human/) and [how to score handoff quality](/blog/human-handoff-quality-score-ai-agents/).

### Silent-failure rate

```text
silent_failure_rate = conversations judged failed with no error span / conversations with no error span
```

This is the metric traces can't give you, and the one most worth adding. It covers confident wrong answers, claimed actions that never happened, and ignored instructions. A rise here with flat tool errors usually means a prompt, model, or retrieval change. See [how to detect when an agent claims success without taking action](/blog/detect-ai-agent-false-success/).

### Frustration rate

```text
frustration_rate = conversations with at least one frustration signal / all conversations
```

Signals include rephrasing the same request, "talk to a human," all caps, profanity, and leaving mid-task. Judge them per conversation, not per message, or one angry message counts five times.

### Policy violation rate

Write each policy as one narrow rule, such as "State the cancellation fee before confirming a cancellation." Rules about money, safety, or legal commitments should have targets near zero and alert on a handful of cases, not a rate.

## Reliability metrics: these come from traces

These are the metrics you can measure on every request as it happens, so they suit classic SLOs.

### Tool-call error rate

```text
tool_error_rate = execute_tool spans with error.type set / execute_tool spans
```

Break it down by `gen_ai.tool.name`. A 1% overall rate can hide one tool failing 20% of the time. Also count tool calls that "succeeded" with empty results, which the agent then papers over.

### Latency p95 and time to first token

Measure latency per user turn, from the user's message to the last assistant token, not per model call. An agent turn may make several model and tool calls. For streaming chat, time to first token is what the user feels. OpenTelemetry defines `gen_ai.server.time_to_first_token` for model servers. For an agent, record your own client-side histogram from user message to first streamed token.

### Loop rate

```text
loop_rate = conversations with 3+ identical tool calls (same tool, same arguments) or a repeated answer / all conversations
```

Identical tool calls are visible in traces. Repeated answers need the text. OpenTelemetry's GenAI conventions now define `gen_ai.invoke_agent.tool_calls`, a per-run tool-call count, which is a useful early signal. See [how to find repeated failed attempts](/blog/repetition-red-flag-looping-conversations/).

## Cost per resolved conversation

```text
cost_per_resolved = (model cost + tool and API cost + infrastructure + human handoff cost) / resolved conversations
```

Divide by resolved conversations, not all of them. A cheaper model that resolves less then shows up as more expensive, which is the truth. Token costs come from traces. Resolution comes from the judge. The full breakdown is in [measure AI agent cost per completed task](/blog/ai-agent-cost-per-completed-task/).

## User signals

- **Thumbs-down rate** on assistant messages. Low volume and biased toward angry users, but cheap and direct.
- **CSAT on AI-handled conversations.** Join survey results to conversations by session ID, then break CSAT down by intent and by failure type to see which failures cost you.
- **Repeat contact within 72 hours** about the same issue. The strongest check on resolution and containment.
- **Abandonment.** The user left mid-task with no resolution.

## How to measure each with OpenTelemetry and evals

Instrument once so both traces and conversation judges can use the same data. Put a stable conversation ID and user ID on every span, follow the GenAI conventions for model and tool spans, and send outcome events (escalation, survey score) with the same IDs.

```python
from opentelemetry import trace

tracer = trace.get_tracer("support-agent")

def call_tool(conversation_id: str, user_id: str, name: str, args: dict):
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        span.set_attribute("gen_ai.conversation.id", conversation_id)
        span.set_attribute("user.id", user_id)
        try:
            return TOOLS[name](**args)
        except Exception as exc:
            span.set_attribute("error.type", type(exc).__name__)
            span.set_status(trace.StatusCode.ERROR)
            raise
```

The GenAI conventions are still marked Development, so attribute names can change. Prefer your framework's built-in instrumentation where it exists. Our [OpenTelemetry guide](/blog/instrument-ai-agent-opentelemetry/) covers setup.

Then split the work:

- **From traces** (your tracing tool or metrics backend): tool error rate, turn latency, time to first token, identical-tool-call loops, tokens and cost.
- **From conversation judges** (online evals): task success, resolution, escalation quality, silent failures, frustration, policy violations, repeated answers.
- **Joined by conversation ID:** silent-failure rate (judge verdict + no error span), cost per resolved conversation (cost + verdict), CSAT by failure type.

## How to alert on each

### Burn-rate alerts for reliability SLOs

Burn rate is "how fast, relative to the SLO, the service consumes the error budget." With a 99% tool-success SLO, the budget is 1% of calls. If 14.4% of calls fail over an hour, you are burning budget at 14.4 times the sustainable rate.

Google's SRE workbook recommends three alerts, each requiring both a long and a short window to be over the threshold:

| Error budget consumed | Long window | Short window | Burn rate | Action |
|---|---|---|---|---|
| 2% | 1 hour | 5 minutes | 14.4 | Page |
| 5% | 6 hours | 30 minutes | 6 | Page |
| 10% | 3 days | 6 hours | 1 | Ticket |

Use these for tool-call success, turn latency, and time to first token. They need volume: at a few hundred conversations a day, one bad hour is a handful of calls, so start with ticket-only alerts until traffic grows.

### Threshold alerts for judged quality metrics

Burn-rate math assumes you see each event almost as it happens. Judged metrics arrive late, because the judge waits for the conversation to end. Alert on them differently:

1. **Use longer windows.** 6 to 24 hours, compared with a 7-day baseline.
2. **Require a minimum count.** Don't alert on 2 failures out of 5 conversations.
3. **Alert on counts for zero-tolerance rules.** A refund-policy violation should notify after 3 cases, not after the rate moves.
4. **Give every alert an owner and a link to the flagged conversations.** Read them before changing the threshold.

### Where Agnost AI fits

Agnost AI covers the conversation-level half. It groups traffic by user and session and judges whole conversations for failures, user frustration, intents, tool-call errors, loops, and silent wrong answers. Online evals are plain-language rules: once a conversation has been idle 15 minutes, an LLM judges it in a batch and marks each assistant message pass or fail against every rule, with no sampling. An alert fires when a rule's violation count crosses a threshold in a lookback window, and goes to Slack or email. Intents let you break each metric down by what users were trying to do. When an alert fires, Auto Improve opens a pull request with a fix tested against past runs, for a person to review, and re-checks the alert after merge.

What it doesn't do: Agnost has no cost tracking, so take cost per resolved conversation from your traces and billing. Judging is batched, not real time, so latency and tool-error burn-rate alerts belong in your metrics stack. Verdicts are pass or fail, not scores.

## Tools that measure these metrics


| Tool | Reliability metrics from traces | Conversation-level judging | Alerting | Cost data |
|---|---|---|---|---|
| **Agnost AI** | Tool-call errors and loops | Every conversation, 15 min idle, no sampling | Rule violation count to Slack or email | No |
| **Langfuse** | Yes | Observation-level judges, with sampling | Aggregated score over a window to Slack, webhook, or PagerDuty | Yes, on generations |
| **LangSmith** | Yes | Thread evaluators, 10 min idle default | Not covered here | Not covered here |
| **Arize AX** | Yes | Session-level evals, with sampling | Not covered here | Not covered here |
| **Braintrust** | Yes | Trace and group scorers, 30 s idle default, with sampling | Not covered here | Not covered here |
| **Datadog** | Error rates, latency, token trends | Goal Completeness on sessions marked complete | Datadog monitors | Token tracking |


### Agnost AI

Conversation-level metrics, as described above. Best for task success, silent failures, frustration, policy violations, and loops on every conversation. Pair it with a tracing or metrics tool for latency, burn-rate SLOs, and cost.

### Langfuse

Open-source tracing with usage and cost captured on generation and embedding observations, so tool errors, latency, and cost per conversation are straightforward. LLM judges target observations, with sampling. Score alerts evaluate an aggregate over a time window, such as average helpfulness below 0.7 over the last hour, and notify Slack, a webhook, or PagerDuty. There is no per-score notification.

### LangSmith

Thread evaluators judge whole conversations after an idle period (10 minutes by default), which suits task success and frustration. Limits are 10 multi-turn evaluators per workspace and 500 threads per run.

### Arize AX

Online evals run at span, trace, or session level. Arize suggests sampling 10-50% for high-volume apps, so rare zero-tolerance failures can be missed unless you judge everything.

### Braintrust

Scoring rules run at span, trace, or group scope with sampling. Trace and group scorers wait 30 seconds of idle by default, which is short for chat where users pause between turns. Strong for turning failed conversations into release tests.

### Datadog Agent Observability

The Operational Insights dashboard shows error rates, latency, token trends, and triggered monitors, and trace metrics are kept 15 months. Goal Completeness judges sessions, but only sessions marked as completed. Good for reliability SLOs if you already run Datadog.

## How to choose your first metrics

Don't start with all thirteen. Start with five:

1. **Tool-call success SLO** with burn-rate alerts. It catches outages.
2. **p95 turn latency** (or time to first token for streaming). It catches slowdowns.
3. **Task success rate**, judged on every conversation. Your headline number.
4. **Silent-failure rate.** The failures nobody else will report to you.
5. **Frustration rate.** Your early warning for churn.

Add resolution, containment, and escalation quality if the agent does support. Add cost per resolved conversation once you're choosing between models. Revisit targets monthly: once a target is met for a month, tighten it or replace it with one that isn't.

## Related reading

- [How to measure AI agent task success without inflating the number](/blog/measure-agent-task-success-rate/)
- [Why is my AI agent's success rate high but users are still unhappy?](/blog/agent-success-rate-vs-agent-experience/)
- [Measure AI agent cost per completed task, not just tokens](/blog/ai-agent-cost-per-completed-task/)
- [How to set up online evals for AI agents in production](/blog/how-to-set-up-online-evals-production/)
- [Best observability platform for a customer-facing AI support agent](/blog/best-observability-customer-support-ai-agent/)
- [Which AI agent observability platform should a startup use?](/blog/ai-agent-observability-for-startups/)

## Frequently asked questions

### What KPIs should I track for a customer-facing AI agent?

Track four groups. Quality: task success rate, resolution rate, containment, escalation rate, silent-failure rate, and frustration rate. Reliability: tool-call error rate, p95 turn latency, time to first token, and loop rate. Cost: cost per resolved conversation. User signals: thumbs-down rate, CSAT on AI-handled conversations, and repeat contact within 72 hours. Quality metrics need each whole conversation judged; reliability metrics come straight from traces.

### What SLOs should I set for an AI agent?

Set classic SLOs only on metrics you can measure on every request in near real time: tool-call success (for example 99% over 30 days), p95 turn latency, and time to first token. Set quality targets, such as task success or silent-failure rate, as weekly objectives with threshold alerts, because they depend on an LLM judge that runs after the conversation ends. Start from a two-week baseline and tighten from there.

### How do you calculate AI agent task success rate?

Task success rate = conversations judged successful / eligible conversations judged, over a fixed window. Define success per task in writing (for example 'the order was cancelled and the user was told the refund amount'), exclude conversations with no real task, and count the judge's verdicts on every conversation, not a sample, if you can afford it. Report unknown outcomes separately instead of counting them as successes.

### What is a good containment rate for an AI support agent?

There is no universal number, and containment alone is misleading. A conversation that ends without a handoff can still be a customer who gave up and called. Measure containment as conversations with no escalation and no repeat contact about the same issue within 72 hours, and watch it next to frustration and resolution. Set the target from your own baseline by intent, since password resets and billing disputes contain very differently.

### How do I alert on AI agent quality without too much noise?

Use multiwindow burn-rate alerts for high-volume reliability SLOs: Google's SRE workbook pages when 2% of a 30-day error budget burns in an hour (burn rate 14.4) and tickets at 10% over three days. For judge-based quality metrics, alert on a count or rate over a longer window (6 to 24 hours) with a minimum number of conversations, route it to a named owner, and review flagged conversations before changing the threshold.

### Which AI agent metrics can I get from OpenTelemetry traces?

Tool-call error rate (execute_tool spans with error.type set), latency per turn and per model call, time to first token if you record it, token usage, and loop signals such as the same tool called with the same arguments repeatedly. Task success, resolution, silent failures, frustration, and policy violations need the conversation text judged, by a person or an LLM, because every span in a failed conversation can be green.

### How do I calculate cost per resolved conversation?

Cost per resolved conversation = (model cost + tool and API cost + infrastructure + human handoff cost) / resolved conversations, for the same window. Divide by resolved conversations, not all conversations, so a cheaper model that resolves less shows up as more expensive. Token costs come from traces; resolution comes from judged conversations.

### How have AI agent KPIs changed since 2025?

Resolution got stricter. Support platforms moved from counting 'no follow-up' as resolved toward LLM-verified resolution: Zendesk split resolutions into Contained and Verified tiers on May 18, 2026. Silent-failure rate, which needs a conversation judge, is the metric most worth adding. And OpenTelemetry's GenAI conventions, now in their own repository, define agent-run metrics such as tool calls per run, which helps with loop detection.
