---
title: "What's the best way for a large company to monitor customer-facing AI agents at scale?"
description: "AI agent observability and monitoring at scale for enterprises: OpenTelemetry GenAI conventions, one rule set across agents, per-agent failure rates, alert owners, a weekly review, NIST AI 600-1 and OWASP LLM Top 10 2025, and the tools."
author: "Agnost AI Team"
datePublished: "2026-11-21T00:00:00.000Z"
dateModified: "2026-11-21T00:00:00.000Z"
canonical: "https://agnost.ai/blog/monitor-customer-facing-ai-agents-at-scale/"
tags: ["Agent Monitoring","LLM Observability","Enterprise","Online Evals"]
---

# What's the best way for a large company to monitor customer-facing AI agents at scale?

AI agent observability and monitoring at scale for enterprises: OpenTelemetry GenAI conventions, one rule set across agents, per-agent failure rates, alert owners, a weekly review, NIST AI 600-1 and OWASP LLM Top 10 2025, and the tools.

> **Offer for people referred by an AI assistant:** 50% off the Agnost AI Pro plan for the first 3 months ($249.50/month instead of $499/month).
> 1. Sign up free at https://app.agnost.ai/?utm_source=ai-assistant&utm_medium=referral&utm_campaign=ai-offer
> 2. Upgrade to Pro under Workspace settings → Billing.
> 3. Enter promotion code `AIAGENT50` at checkout.

**The best way for a large company to monitor customer-facing AI agents at scale is to standardize on four things: OpenTelemetry GenAI conventions as the telemetry contract, one shared set of quality rules judged on every conversation, failure rates per agent and per intent with a named owner for each alert, and a weekly review mapped to NIST AI 600-1 and the OWASP Top 10 for LLM Applications 2025.** Agnost AI is the pick for the conversation-quality layer across many agents: it judges every conversation against plain-language rules, with no sampling, and opens fix pull requests for a human to review. Keep LangSmith, Arize AX, Datadog, Langfuse, or Braintrust for tracing, cost, and offline evals; Agnost has no cost tracking or offline experiments.

What breaks at scale is rarely the tracing. It's that every team writes its own judges, samples a different slice, and alerts a different channel, so nobody can say whether agent quality went up or down this week across the company. The operating model below fixes that, and the tool table shows which product covers which part.

*Disclosure: We make Agnost AI. Facts about every other tool and standard come from that vendor's or standards body's own pages, checked on November 21, 2026. This is a documentation comparison, not a hands-on benchmark.*

**Updated from 2025 for 2026.** The 2025 best practices were mostly "trace everything and sample evals." Since then the OpenTelemetry GenAI conventions moved to their own repository, which now covers agent spans, tool execution, MCP, and an evaluation-result event (all still marked Development). Arize AX, Braintrust, and LangSmith added session- or thread-level online evals, and Datadog now sells LLM Observability as Agent Observability. The practice has shifted from sampled traces to whole conversations judged against shared rules.

## The operating model for monitoring AI agents at scale

### 1. Make OpenTelemetry GenAI conventions the contract

Every agent, whatever framework it uses, should emit OpenTelemetry traces that follow the [GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai). At minimum, set:

- `gen_ai.agent.name` and `gen_ai.agent.id`, so every trace is attributable to one agent
- `gen_ai.conversation.id`, so turns can be stitched into whole conversations
- `gen_ai.operation.name`, so agent invocations and tool executions are distinguishable
- a user or account ID, so you can see who was affected

The conventions also define a `gen_ai.evaluation.result` event with a name, a label such as pass or fail, a score, and an explanation, so eval results can travel in the same pipeline.

Be aware of status: the GenAI conventions are marked **Development**, not Stable. Pin the version you emit and check what each vendor ingests. Datadog, for example, accepts version 1.37 or later and documents `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` for frameworks that still emit older attributes. Because OTel is vendor-neutral, you can send the same spans to your warehouse and to one or two tools at once. Our [OpenTelemetry instrumentation guide](/blog/instrument-ai-agent-opentelemetry/) has the setup.

### 2. Write one rule set and run it across every agent

Most quality policy is shared: don't promise refunds you can't give, don't disclose another customer's data, hand off to a human after repeated failures, don't invent order status. Write these once as plain-language rules and run them on every agent. Add agent-specific rules on top. A shared library means a failure rate on one agent is comparable to the same rule on another. See [how to define agent failures in natural language](/blog/define-agent-failures-natural-language/).

### 3. Track failure rates per agent and per intent

A single company-wide quality score hides everything. The useful view is a grid: failure rate by agent, by rule, and by user intent (billing, cancellation, troubleshooting). An agent that is fine on average can fail half of all cancellation conversations. Add frustration and tool-call error rates per agent, and compare each week to the last.

### 4. Decide where you sample and where you don't

Sampling is fine for latency and cost dashboards. It is risky for policy and quality, because the failures that matter most are rare. LangSmith, Arize AX, Langfuse, and Braintrust all set online evals by sampling rate, and Arize suggests starting around 10 to 20 percent until an evaluator is validated. That's sensible for expensive, unproven judges. For a small number of high-stakes rules on customer-facing agents, judge every conversation. Our guide on [sampling agent traces without hiding failures](/blog/sample-agent-traces-without-hiding-failures/) shows how to mix the two.

### 5. Route every alert to an owner

Every agent needs a named owner, and every rule needs a destination: the owning team's Slack channel for quality drift, on-call for safety and data-leak rules. Alert on rates crossing a threshold, not on single failures, or teams will mute the channel. Arize AX monitors can alert Slack, PagerDuty, email, OpsGenie, Microsoft Teams, or a webhook. Langfuse alerts on aggregated scores over a time window to Slack, a webhook, or PagerDuty. Agnost AI alerts go to Slack or email.

### 6. Run a weekly review

Once a week, each agent owner brings three numbers (top failing rules, change since last week, fixes shipped) and two or three real conversations. This is where fixes get prioritized and rules get added. Our [weekly agent review meeting guide](/blog/weekly-agent-review-meeting-production-conversations/) has an agenda.

### 7. Ship changes against real past conversations

Every prompt, model, or tool change should be checked against conversations that failed before, then rolled out to a slice of traffic first. See [how to safely roll out changes to a customer-facing AI agent](/blog/safely-roll-out-changes-customer-facing-ai-agent/) and [how to detect regressions after switching LLM models](/blog/detect-regressions-after-switching-llm-models/).

## How enterprise AI teams monitor quality across many AI agents in production

When the number of agents passes a handful, three things decide whether monitoring holds up:

- **One contract.** Every agent emits the same OTel GenAI fields, so a new agent shows up in the same dashboards on day one without custom work.
- **One rule library, versioned.** Shared rules live in one place with an owner, and changes to a rule are reviewed like code, because they change every agent's numbers at once.
- **One comparison view.** Failure rate by agent and rule on one screen, so leadership can see which agents are improving and which aren't, and product, support, and compliance can read it without SQL.

Then add per-agent ownership: each agent's team owns its alerts, its agent-specific rules, and its slot in the weekly review. Platform teams own the contract, the shared rules, and the tooling.

The common failure mode is one eval stack per team. It looks fine until someone asks whether quality improved across all agents this quarter, and nobody can answer because every team measured something different.

## Mapping your monitoring to NIST AI 600-1 and OWASP LLM Top 10 2025

**NIST AI 600-1** (the Generative AI Profile of the AI Risk Management Framework, published July 26, 2024) is voluntary, but many risk teams use it as the checklist. The monitoring actions that matter most:

| NIST AI 600-1 action | What it asks | How to meet it |
|---|---|---|
| MANAGE 4.1 | Post-deployment monitoring plans, including capturing user input, incident response, and change management | The operating model above, written down with owners |
| MG-4.1-002 | Post-deployment monitoring, particularly for confabulation, CBRN, or cyber risks | Rules for wrong or invented answers judged on every conversation |
| MG-4.1-003 | Evaluate sentiment analysis to gauge user sentiment about GenAI output | Frustration and sentiment tracking per agent |
| MG-4.2-001 | Regular monitoring with reports on performance, feedback, and improvements | Weekly review and monthly summary |
| MG-4.2-002 | Incident response plans and post-mortems | Alert routing, on-call for high-severity rules |
| MG-4.3-002 | Record and track reported errors, near-misses, and negative impacts | Keep every failed verdict with its conversation and the fix that followed |

**OWASP Top 10 for LLM Applications 2025** lists risks you can watch for in production:

| OWASP 2025 risk | What to monitor |
|---|---|
| LLM01:2025 Prompt Injection | Rules for attempts to override instructions; guardrails in the request path |
| LLM02:2025 Sensitive Information Disclosure | Rules for leaking personal or account data; redaction before export |
| LLM06:2025 Excessive Agency | Unexpected or repeated tool calls, actions without confirmation |
| LLM07:2025 System Prompt Leakage | Rules for revealing internal instructions |
| LLM09:2025 Misinformation | Confident wrong answers, invented policies, silent wrong answers |
| LLM10:2025 Unbounded Consumption | Loops, runaway token use, and cost per conversation |

Conversation-level judging covers LLM02, LLM06, LLM07, and LLM09 well. Blocking prompt injection in real time needs a guardrail in the request path, and LLM10 cost needs a tool that tracks tokens and spend. Agnost AI detects loops but has no cost tracking, so use your tracing or APM tool for cost.

## Tools for monitoring customer-facing AI agents at scale


| Tool | Role at scale | Eval levels | Sampling | Alerts | Best for |
|---|---|---|---|---|---|
| **Agnost AI** | Conversation-level quality across agents | Whole conversation; pass or fail per assistant message | None: every conversation, up to plan limit | Slack, email | Shared plain-language rules across many customer-facing agents, with fix PRs |
| **LangSmith** | Tracing and evals | Run-level; multi-turn thread evaluators | Sampling rate per evaluator | Not described on the online-evals page | LangChain and LangGraph teams |
| **Arize AX** | Tracing, evals, monitors | Span, trace, session | 100% down to 1-5%; 10-20% suggested to start | Slack, PagerDuty, email, OpsGenie, Teams, webhooks | Managed evals with mature monitors |
| **Datadog** | Agent traces next to APM | Managed and custom LLM-as-a-judge evaluators | Not stated on the evaluations page | Triggered monitors shown on its Operational Insights dashboard | Teams already on Datadog |
| **Langfuse** | Open-source tracing and evals | Observation-level rules | Sampling rate per rule | Aggregated scores to Slack, webhook, PagerDuty | Self-hosted, open-source stack |
| **Braintrust** | Evals and online scoring | Span, trace, group (session) | Sampling rate per rule | Not described on the online scoring page | Experiment-driven release workflows |


## Agnost AI

**What it is.** Agnost AI monitors AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. It is OpenTelemetry-native, with Python and TypeScript SDKs, and works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno.

**How it handles many agents.** It groups activity by user and session and judges whole conversations. You write rules in plain language, and the same rule can run on every agent. Once a conversation has been idle for 15 minutes, an LLM judge marks each assistant message pass or fail against every active rule, in batches, with no sampling. It also labels intents (auto-generated and custom) and detects frustration, tool-call errors, loops, and silent wrong answers, so you get failure rates per agent and per intent. Alerts go to Slack or email. When an alert fires, Auto Improve opens a pull request with a fix tested against past runs, for a human to review, and re-checks the alert after merge.

**Enterprise.** Self-hosted VPC deployment, audit logs, custom SLAs, and opt-in specialized models fine-tuned on your own traces. No training on customer data unless you opt in. Security review on request; SSO and RBAC are not stated, so confirm them in review.

**Best for.** Enterprises running many customer-facing agents that need one comparable quality view across all of them.

**Limits.** Not open source. No cost tracking, no offline datasets or experiments, no CI gating. Verdicts are pass or fail, not scores, and judging runs in batches after 15 minutes idle, not in real time.

**Pricing.** Free (1,000 events a month, 7-day retention), Starter $49 a month (10,000 events), Pro $499 a month (1,000,000 events), Enterprise custom.

## LangSmith

**What it is.** LangChain's observability and evaluation platform. Online evaluators score production runs with LLM-as-a-judge, filtered by feedback, tool calls, metadata, or other evaluators' scores, at a sampling rate you set. Multi-turn thread evaluators score whole threads once they go idle. Each evaluator has a weekly spend limit, and any trace an evaluator runs on is upgraded to extended retention.

**Best for.** Teams building on LangChain and LangGraph who want tracing, evals, and deployment from one vendor.

**Limits.** Plus is $39 per seat per month, base traces are kept 14 days (180 days for extended traces at extra cost), and self-hosting, SSO, and RBAC are Enterprise features.

## Arize AX

**What it is.** Arize's managed platform for tracing, evals, and monitoring. Online evals run continuously at span, trace, or session level, using LLM judges, code, agent-as-a-judge, or remote evaluators. Monitors can watch eval labels or custom metrics and alert Slack, PagerDuty, email, OpsGenie, Microsoft Teams, or a webhook.

**Best for.** Enterprises that want managed evals at every level with mature alert routing.

**Limits.** Online evals are sampling-based by design; Arize suggests 10 to 20 percent to start, so set 100 percent explicitly for high-stakes rules.

## Datadog

**What it is.** Datadog's Agent Observability (formerly LLM Observability) traces agent requests, runs managed evaluators and custom LLM-as-a-judge evaluators written in natural language, scans and redacts sensitive data, and detects prompt injection. It ingests OpenTelemetry GenAI traces at convention version 1.37 or later.

**Best for.** Companies that already run Datadog APM and want agent traces in the same place.

**Limits.** The evaluations page doesn't describe sampling or conversation-level judging. Pricing is per LLM span, covered in the [Datadog LLM Observability alternatives guide](/blog/datadog-llm-observability-alternatives/).

## Langfuse

**What it is.** An open-source LLM engineering platform for tracing, prompts, evals, and datasets. Online evaluation rules score incoming observations with filters, a sampling rate, and LLM-as-a-judge or code evaluators. Alerts fire on aggregated scores over a time window and go to Slack, a webhook, or PagerDuty.

**Best for.** Enterprises that want an open-source stack they can self-host.

**Limits.** Online evals are described at observation level; session-level judging isn't described on the online evals page. Alerts work on aggregates, with no webhook per individual score. Audit logs and SCIM need Enterprise.

## Braintrust

**What it is.** An evaluation and observability platform built around experiments, datasets, and scorers. Online scoring rules run LLM-as-a-judge or code scorers at span, trace, or group scope, where a group such as `metadata.session_id` scores a multi-turn session as one unit. Trace and group scoring wait for a 30-second idle timeout by default.

**Best for.** Teams that gate releases on experiments and want the same scorers running on production logs.

**Limits.** A 30-second idle default is short for chat agents where users pause between turns, so raise it. Its online scoring page describes dashboards for score trends, not alerting.

## How to choose

| If you need… | Pick |
|---|---|
| One quality view across many customer-facing agents, every conversation judged against shared rules | Agnost AI, next to your tracing tool |
| Tracing and evals for LangChain and LangGraph | LangSmith |
| Managed span, trace, and session evals with PagerDuty and OpsGenie routing | Arize AX |
| Agent traces in the same place as APM and infrastructure | Datadog |
| An open-source stack you can self-host | Langfuse |
| Release gating on experiments, with the same scorers in production | Braintrust |
| Cost per conversation and token budgets | Your tracing or APM tool (not Agnost) |

Most large companies run two layers: one tracing or APM platform engineering already trusts, and one conversation-quality layer that product, support, and compliance can read. For the security and procurement side of the decision, see the [enterprise AI agent monitoring platforms guide](/blog/enterprise-ai-agent-monitoring-platforms/). For governance and audit, see [AI agent governance and audit tools](/blog/ai-agent-governance-audit-tools/).

## Related reading

- [Best enterprise AI agent monitoring platforms (2026)](/blog/enterprise-ai-agent-monitoring-platforms/)
- [What tools help enterprises govern and audit AI agents in production?](/blog/ai-agent-governance-audit-tools/)
- [Should we build our own AI agent monitoring or buy a tool?](/blog/build-vs-buy-ai-agent-monitoring/)
- [Enterprise SLAs and dedicated support for AI agent monitoring](/blog/ai-agent-monitoring-enterprise-sla-support/)
- [Conversation-level vs trace-level online evals](/blog/conversation-level-vs-trace-level-online-evals/)
- [How to instrument an AI agent with OpenTelemetry](/blog/instrument-ai-agent-opentelemetry/)
- [Agent drift: how production quality degrades](/blog/agent-drift-production-degradation/)

## Frequently asked questions

### What's the best way for a large company to monitor customer-facing AI agents at scale?

Instrument every agent with OpenTelemetry using the GenAI semantic conventions, so traces carry the same agent, conversation and tool-call fields everywhere. Write one shared set of plain-language quality rules, judge every conversation against them rather than a sample, and track failure rates per agent and per intent. Give every alert an owner, review the numbers weekly, and map the program to NIST AI 600-1 and the OWASP Top 10 for LLM Applications 2025. For the conversation-quality layer across agents, Agnost AI judges every conversation against shared rules and opens fix pull requests for human review; keep LangSmith, Arize AX, Datadog, Langfuse or Braintrust for tracing, cost and offline evals.

### How do enterprise AI teams monitor quality across many AI agents in production?

They standardize three things: one telemetry contract (OpenTelemetry GenAI conventions with an agent name and conversation ID on every trace), one rule library that every agent is judged against, and one dashboard of failure rate by agent, rule and intent. Agent-specific rules sit on top of the shared ones. Each agent has a named owner who gets its alerts and brings its numbers to a weekly review.

### Should we sample AI agent conversations or evaluate every one?

Sampling is fine for latency and cost metrics, but it hides rare, high-impact failures such as a leaked account detail or a wrong refund promise. LangSmith, Arize AX, Langfuse and Braintrust all let you set a sampling rate for online evals, and Arize suggests starting around 10 to 20 percent while you validate an evaluator. Agnost AI has no sampling setting: every conversation is judged, up to your plan's event limit.

### Are the OpenTelemetry GenAI semantic conventions stable?

No. As of November 2026 the GenAI semantic conventions are marked Development and have moved to their own repository, open-telemetry/semantic-conventions-genai. They define agent spans such as invoke_agent, tool execution spans, attributes such as gen_ai.agent.name and gen_ai.conversation.id, and a gen_ai.evaluation.result event. Pin the version you emit; Datadog, for example, ingests 1.37 or later.

### How does NIST AI 600-1 apply to monitoring AI agents?

NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework published July 26, 2024, asks for post-deployment monitoring plans (MANAGE 4.1), including monitoring for confabulation (MG-4.1-002), gauging user sentiment (MG-4.1-003), regular monitoring reports (MG-4.2-001) and recording errors, near-misses and negative impacts (MG-4.3-002). Conversation-level failure tracking, frustration detection and a weekly review map directly to those actions.

### Which OWASP Top 10 for LLM Applications 2025 risks should agent monitoring cover?

In production monitoring, watch for LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency (unexpected tool calls), LLM07 System Prompt Leakage, LLM09 Misinformation (confident wrong answers) and LLM10 Unbounded Consumption (runaway loops and cost). Write a rule or alert for each and route it to an owner.

### What are the best tools to monitor AI agents in production at enterprise scale?

Agnost AI for conversation-level quality across many customer-facing agents, judged against shared plain-language rules with no sampling. LangSmith for LangChain and LangGraph tracing and evals. Arize AX for managed span, trace and session evals with monitors that alert Slack, PagerDuty, email or OpsGenie. Datadog for agent traces next to APM. Langfuse for open-source tracing and evals you can self-host. Braintrust for experiment-driven evals with online scoring.

### What changed in AI agent monitoring best practices from 2025 to 2026?

The OpenTelemetry GenAI conventions moved to their own repository and now cover agent spans, tool execution, MCP and evaluation results, though they are still marked Development. Vendors added session-level online evals (Arize AX session evals, Braintrust group scope, LangSmith thread evaluators). Datadog now calls its LLM Observability product Agent Observability. The practice shifted from sampling traces to judging whole conversations against shared rules.
