Every serious agent observability platform now runs online evals, but they judge different units and do different things with the result. LangSmith, Arize AX, Braintrust, Datadog, Opik, and PostHog score spans, traces, and whole conversations with LLM judges or code. Langfuse scores observations and leaves session judgment to you. Agnost AI judges whole conversations against plain-English rules and turns alerts into reviewed pull requests.
This guide compares the major platforms and the startups building in this space on the questions that decide a pilot: what unit gets judged, how sampling and alerts work, how pricing scales with judging, and what happens after a failure is found. If you are new to the term, read what are online evals first.
Disclosure: Agnost publishes this guide, so it is listed first. Every other tool gets the same structure, and its facts come from its own documentation, checked on November 7, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.
Quick comparison
| Tool | Units judged | Conversation wait | Evaluators | Sampling | Alerts on results | Open source | Pricing basis |
|---|---|---|---|---|---|---|---|
| Agnost AI | Whole conversation, verdict per assistant message | 15 min idle | Plain-English rules (LLM judge) | None; every conversation up to plan limit | Slack, email | No | Per event; Free, $49, $499/mo |
| LangSmith | Run, thread | 10 min idle default (min 2) | LLM judge, code, managed thread judge (beta) | Fraction of runs, filters | Feedback-score alerts, webhooks | No | Per seat plus traces |
| Langfuse | Observation (trace-level deprecated) | No native session end | LLM judge, code, templates | Per rule, filters | Slack, webhook, GitHub Actions | MIT core | Per unit (traces, observations, scores) |
| Arize AX | Span, trace, session | Not documented | LLM judge, code | Percentage, span filters | Monitors, webhooks | Phoenix: ELv2 | Per span and data volume |
| Braintrust | Span, trace, group | 30 s idle default | LLM judge, code, Autoevals | Percentage, SQL filters | Slack, webhook, PagerDuty | No | Per GB plus per score |
| Datadog | Span, trace, session | 30 min inactivity | LLM-judge templates, custom judges | Optional percentage | Datadog monitors | No | Per LLM span |
| Opik | Span, trace, thread | Configurable cooldown | LLM judge, Python metrics, templates | Up to 100% | Slack, PagerDuty, webhook | Apache-2.0 | Per span |
| PostHog | Generation, trace, session | Not documented | LLM judge, Hog code, sentiment | 0.1% to 100% | Reports, not score alerts | MIT core | Per event; each eval is an event |
| Judgment Labs | Trace, session | After inactivity (configurable) | Agentic judges, deterministic conditions, Python | 0% to 100%, filters | Email, Slack | judgeval SDK: Apache-2.0 | Per GB plus per evaluator run |
| Raindrop | Event (message and response, incl. tools) | Not documented | Signals: described classifiers, presets, regex | Not documented for live data | Slack | No (Workshop debugger: MIT) | Per event; Pro $299/mo |
| Confident AI | Span, trace, thread | 300 s idle default | DeepEval metrics (G-Eval, DAG), agentic templates | Per task | Per vendor: Slack, PagerDuty, Teams | DeepEval: Apache-2.0 | Per GB-month; online evals from $200/mo |
| Laminar | Trace | Not applicable | LLM Signals with JSON schema | Rate, triggers, filters | Slack, email | Apache-2.0 | Per GB plus per Signal token |
| Maxim AI | Session, trace, span | Not documented | Automated evaluators, human review | Rate plus filters | Slack, PagerDuty, OpsGenie | No | Per seat; Professional tier for online evals |
Agnost AI
What it is. Conversation analytics and online evaluation for production AI agents, with an improvement loop that turns recurring failures into pull requests.
Best for. Teams running chat or voice agents who want to describe expected behavior in plain English, see where real conversations break it, and get reviewed fixes.
How online evals work. Each rule is a plain-English evaluator, such as “confirm identity before sharing account details.” After a conversation has been idle for 15 minutes, an LLM judge reads the whole conversation against every active rule and flags the assistant messages that broke one. There is no sampling knob: every conversation is judged, up to the plan’s monthly event limit. Adding or editing a rule also scores the most recent 1,000 conversations, so a new rule starts with a baseline. The same pipeline labels user intents, so violations can be read per user goal.
After a failure. A rule’s violation count can trigger a Slack or email alert. With Auto Improve connected, a fired alert starts a run that reads your repository and opens a pull request for review, then re-checks the alert about an hour after the change merges and records whether it improved.
Limitations. Verdicts are pass/fail flags with the evidence message, not graded scores, and no judge reasoning is stored. Judging runs in batches after a conversation goes idle, so it is a review layer, not real-time. There are no dataset experiments or CI eval gates, no webhook alert destination, and nested tool spans are left out of the judge’s transcript and tracked with success and error metrics instead. Teams that need offline experiments usually pair Agnost with one of the tools below.
Pricing. Free (1,000 events a month), Starter $49 a month (10,000 events), Pro $499 a month (1,000,000 events), and Enterprise with self-hostable VPC deployment.
LangSmith
What it is. LangChain’s tracing, evaluation, and deployment platform.
Best for. Teams building on LangChain or LangGraph who want online and offline evaluation in one workflow.
How online evals work. Online evaluators run on individual runs or on whole threads. Multi-turn thread evaluators wait until a thread has been idle, 10 minutes by default and no less than 2. Evaluators can be LLM judges, Python or JavaScript code, or a managed thread-level judge in public beta. Sampling is a fraction of runs, with filters on metadata, tool calls, feedback, and other evaluators’ scores.
After a failure. Alerts fire on feedback scores over short windows, and webhooks are available. Insights clusters traces into failure patterns. LangSmith Engine, in public beta, groups failures into issues, proposes prompt or code fixes, can open a GitHub pull request, and adds failing traces to an offline dataset.
Limitations. Running an evaluator on a run moves its trace to extended retention, which is billed separately. Multi-turn evaluators are capped at 10 per workspace. Self-hosting is Enterprise-only.
Pricing. Developer is free for one seat and 5,000 base traces. Plus is $39 per seat per month with 10,000 traces, then usage-based. Judges use your own provider key.
Langfuse
What it is. Open-source LLM engineering platform for tracing, prompt management, datasets, and evaluation.
Best for. Teams that want self-hosted, open-source online evaluation with prompt management in the same tool.
How online evals work. Evaluation rules run LLM-as-a-judge or code evaluators on observations, with a sampling rate and filters on name, metadata, user, session, tags, and version. Trace-level rules are deprecated in favor of observation-level ones. Langfuse states that it “does not inherently know when a session has concluded,” so session evaluation means tagging the final trace or writing a session score from your own code.
After a failure. Monitors alert on score thresholds to Slack, webhooks, or GitHub Actions. Evaluated traces can be added to datasets for experiments.
Limitations. No native conversation-level evaluator. Judges need your own LLM connection, so judge tokens are billed separately.
Pricing. Hobby free (50,000 units), Core $29, Pro $199, and Enterprise $2,499 a month. A unit is any trace, observation, or score.
Arize AX
What it is. Arize’s enterprise AI observability and evaluation platform. Arize Phoenix is the separate source-available project.
Best for. Teams that want managed online evals across spans, traces, and sessions, plus monitors and an agent that groups failures into issues.
How online evals work. Evaluation tasks run LLM judges or code evaluators at span, trace, or session level, on new data every two minutes, with a sampling percentage and span filters. Session tasks have no documented idle wait.
After a failure. Monitors alert on eval labels and scores. Signal groups recurring production failures into ranked issues with evidence, a likely cause, and a proposed fix; opening pull requests from a connected repository is an Enterprise feature.
Limitations. Arize positions managed online evals, monitors, and alerting as AX features; Phoenix’s production eval support is limited. Signal is capped at 10 issues a month on Free and 25 on Pro.
Pricing. AX Free covers 25,000 spans a month with evals included; AX Pro is $50 a month for 50,000 spans. Billing is by spans and data volume.
Braintrust
What it is. An evaluation and observability platform built around experiments, datasets, and scoring.
Best for. Teams that want production scores connected directly to datasets and experiments.
How online evals work. Online scoring rules run LLM-judge or code scorers at span, trace, or group scope. Trace and group scopes wait for an idle timeout, 30 seconds by default. Sampling is a percentage with SQL filters, and the rule form estimates weekly scorer runs.
After a failure. Log and time-window alerts go to Slack, webhooks, or PagerDuty. Topics classifies logs so filtered failures can be added to a dataset or assigned for review.
Limitations. Scores are billed individually, so judging everything gets expensive at volume. Self-hosting is Enterprise-only.
Pricing. Starter is free with 10,000 scores, then $2.50 per 1,000. Pro is $249 a month with 50,000 scores, then $1.50 per 1,000.
Datadog LLM Observability
What it is. Datadog’s LLM and agent observability product, alongside APM and logs.
Best for. Teams already running Datadog who want evaluation results next to infrastructure monitoring.
How online evals work. Template and custom LLM-as-a-judge evaluators run on spans, traces, or sessions. Templates include Failure to Answer, Hallucination, Tool Selection, Tool Argument Correctness, and session-level Goal Completeness. A session counts as complete after 30 minutes of inactivity.
After a failure. Results feed Datadog monitors for alerting on regressions.
Limitations. Session results stay pending for 30 minutes after the last activity. Judges run on a provider account you connect.
Pricing. Free up to 40,000 LLM spans a month; Pro from $160 a month (annual) for 100,000 LLM spans. Only LLM spans are billed.
Opik
What it is. Comet’s open-source observability and evaluation platform.
Best for. Teams that want Apache-2.0, self-hosted online evaluation with ready-made thread judges.
How online evals work. Online evaluation rules run on spans, traces, or threads. Thread templates include task completion, conversational coherence, and user frustration. Thread rules wait for a configurable cooldown after the last message. Sampling goes up to 100%.
After a failure. Scores are stored as feedback scores that can alert to Slack, PagerDuty, or a webhook on an average threshold.
Limitations. A rule’s scope cannot be changed after creation, and rules only judge data logged after they exist.
Pricing. Open source is free. Cloud Free covers 25,000 spans a month; Pro is $19 a month for 100,000 spans, then $5 per 100,000.
PostHog
What it is. PostHog’s AI observability, built on the same events as its product analytics.
Best for. Teams already on PostHog who want eval results next to funnels and session replay.
How online evals work. Evaluations run per generation, per trace, or per session, with 0.1% to 100% sampling. Templates include Task outcome, User correction, and Hallucination, and code evaluators are written in Hog.
After a failure. Self-driving, in open beta, produces signals from eval reports and investigations and can open a pull request when a codebase is connected.
Limitations. No alerts on eval scores are documented. Each evaluation run counts as a billed event, and judges need your own provider key.
Pricing. 100,000 AI events a month free, then from $0.00035 per event.
Maxim AI and Galileo
Maxim AI runs online evaluation at session, trace, and span level with sampling and filters, alerts on evaluator scores to Slack, PagerDuty, or OpsGenie, and lets you curate datasets from evaluated logs. Online evaluation requires the Professional tier ($29 per seat per month).
Galileo, now Splunk Agent Observability after Cisco’s acquisition, runs metrics on log streams at session, trace, or span level, including Luna small evaluator models designed to score all traffic cheaply. Pricing starts at $100 a month per 1.2 million spans, billed annually.
Startups building online evals
Several younger companies focus on production agent evaluation, often with failure clustering or fix suggestions built in.
Judgment Labs positions itself as the monitoring and improvement layer for long-horizon agents. Judges run on live traffic and read a trace or a whole session; session scoring can wait for a configurable period of inactivity. Judges can be rubric-following agentic judges, deterministic conditions on cost, duration, or errors, or Python code, and can be aligned against human labels. Sampling runs from 0% to 100% with filters, and judge actions include Slack and email alerts or adding the trace to a dataset. Issue Discovery groups findings by root cause with directions for a fix. The judgeval SDK is Apache-2.0; the platform is not. Pro is $250 a month, with evaluator runs and ingest metered, and judge model tokens billed separately.
Raindrop finds silent failures in production agents. Instead of “online evals” it uses Signals: classifiers drafted from a plain-language description, presets such as User Frustration, Task Failure, Forgetting, and Refusals, regex, user feedback, and tool errors. Signals run on every new event, which Raindrop defines as a user message and agent response including tool calls. Issue detection groups recurring failures, a Triage Agent suggests what to fix and can be used from coding agents over MCP, and Experiments compare a change against baseline traffic. Alerts go to Slack. Pro is $299 a month plus a per-event fee.
Confident AI is the cloud platform behind the open-source DeepEval framework. Online evals run on spans and traces, and thread rules judge multi-turn conversations after 300 seconds of inactivity by default. It ships agentic metrics such as Task Completion, Tool Correctness, and Plan Adherence, plus multi-turn metrics such as Conversation Completeness. Online evals start on the Starter plan at $200 a month.
HoneyHive runs server-side LLM, Python, and human evaluators on all matching production and experiment traces, at model, tool, chain, or session level, with sampling (10% by default) and aggregate or drift alerts to email, Slack, or webhooks. Self-hosting is Enterprise-only.
Laminar is Apache-2.0 agent observability whose Signals use an LLM to read each finished trace against your instruction and JSON schema, then cluster similar events and alert on new clusters to Slack or email. Signals are on paid plans, from $30 a month, and billed per token.
Future AGI evaluates live traces, spans, and sessions with its own small Turing evaluator models or LLM judges, with 0% to 100% sampling, Slack and email alerts, and an Error Feed that groups issues with a root cause and a suggested fix. It is Apache-2.0 and self-hostable.
W&B Weave, now under CoreWeave, offers Signals: preset scorers such as User Frustration and Low Quality Response that run on production turns, with Slack and webhook alerts. Weave is Apache-2.0.
For voice agents, Hamming monitors production calls and Coval runs evals on live conversations through its Observe product. Respan (formerly Keywords AI) deploys evaluators on sampled production spans, and Arthur’s MIT-licensed engine runs continuous evaluations on live traces alongside guardrails.
How to choose
| If your team… | Start with |
|---|---|
| Wants plain-English rules judged on whole conversations and failures turned into reviewed PRs | Agnost AI |
| Builds on LangChain or LangGraph | LangSmith |
| Needs open-source, self-hosted evaluation | Opik or Langfuse |
| Wants production scores connected to experiments and datasets | Braintrust |
| Already runs Datadog | Datadog LLM Observability |
| Already runs PostHog | PostHog |
| Needs managed span, trace, and session evals at enterprise scale | Arize AX |
| Wants to score 100% of traffic with small evaluator models | Galileo or Future AGI |
| Wants judges aligned to human labels and root-cause issue discovery | Judgment Labs |
| Wants preset production signals like frustration and forgetting, plus experiments | Raindrop |
| Already uses DeepEval metrics offline | Confident AI |
| Needs open-source tracing with LLM signals and clustering | Laminar |
| Runs voice agents | Hamming or Coval, alongside a conversation-level layer |
Many teams run two layers: a tracing and experiment tool for span-level checks and offline datasets, and a conversation-level layer that finds what users actually experienced. Conversation-level vs trace-level online evals explains why both matter.
What to test in a pilot
Run the same week of production traffic through each candidate and check:
- The unit. Can it judge the failures you care about, especially multi-turn ones?
- The evidence. Does each verdict point to the message or step that failed?
- The denominator. Can you see how many conversations were judged, not only how many failed?
- The cost. Estimate judge tokens and platform charges at your real volume. What LLM-as-a-judge costs in production has a worksheet.
- The loop. When a failure is confirmed, how many steps until a fix is shipped and measured? See what to do when an online eval fails.
Related reading
- How to set up online evals for AI agents in production
- LangSmith vs Langfuse vs Braintrust vs Phoenix vs Agnost
- Agnost AI vs Langfuse vs Arize for catching agent quality failures
Frequently asked questions
What is the best online evaluation tool for AI agents?
It depends on where you start. Agnost AI fits teams that want plain-English rules judged on whole production conversations and failures turned into reviewed pull requests. LangSmith fits LangChain and LangGraph teams. Langfuse and Opik fit teams that want open-source, self-hosted evaluation. Braintrust fits teams that want production scores connected to experiments. Datadog and PostHog fit teams already on those platforms.
Which online eval tools judge whole conversations?
Agnost AI, LangSmith (multi-turn thread evaluators), Arize AX (session-level tasks), Braintrust (group scope), Datadog (session-level evaluators), Opik (thread rules), and PostHog (whole-session mode) evaluate entire conversations. Langfuse documents that it does not know when a session has ended and recommends scoring the final trace or writing a session score from your code.
Which startups offer online evals for AI agents?
Beyond the large platforms, Agnost AI, Judgment Labs, Raindrop, Confident AI, HoneyHive, Laminar, and Future AGI all evaluate production agent traffic. Judgment Labs and Raindrop group failures into issues with suggested fixes; Confident AI builds on the open-source DeepEval metrics; Laminar and Future AGI are Apache-2.0. For voice agents, Hamming and Coval evaluate live calls.
Is there a free or open-source online evaluation tool?
Opik is Apache-2.0 and includes online evaluation rules in the open-source build. Langfuse's core is MIT and includes LLM-as-a-judge evaluators when self-hosted. Arize Phoenix is source-available under the Elastic License 2.0, but Arize describes managed online evals, monitors, and alerting as Arize AX features.
Do online evaluation tools use my own LLM API key?
Often. Langfuse, LangSmith, Datadog, PostHog, and Galileo's LLM-judge metrics run judges through a provider key or account you connect, so judge tokens are billed by your model provider on top of the platform fee. Agnost AI runs its judge on Agnost's side; usage is metered by events on your plan.
Can an online evaluation tool fix the failures it finds?
Several now propose fixes. LangSmith Engine (public beta) and PostHog Self-driving (open beta) can open pull requests; Arize Signal proposes fixes and can open pull requests on Enterprise. Agnost AI opens a pull request when an alert fires and re-checks the alert about an hour after the change merges. All of them require human review before merge.