Every tool here has evaluators, so “can it detect a bad answer?” won’t separate them. Three questions will: what unit each one judges (one model call, one trace, or a whole conversation), whether it finds failures you never wrote a check for, and what happens after a failure is found.
We make Agnost AI. Every Langfuse and Arize claim below comes from their own documentation and pricing pages, checked on 2026-11-02 and listed in the sources for this page. This comparison is narrow on purpose: catching quality failures and fixing them. For tracing, prompts, and pricing in general, see Agnost AI vs Langfuse and the five-way comparison with LangSmith, Braintrust, and Phoenix.
Detectors and remediation, side by side
| Agnost AI | Langfuse | Arize AX | |
|---|---|---|---|
| Unit that gets judged | The whole conversation, once it has been idle for 15 minutes; nested tool spans are tracked separately | One observation (a span or generation); trace-level evaluators are deprecated | A span, a trace, or a session, chosen per eval task |
| How you define a failure | Rules in plain language (“confirm identity before sharing account details”), plus intents for what users wanted | LLM-as-a-judge, code, and decision-model evaluators; human annotation queues | LLM judges, code evaluators, agent-as-a-judge, remote evaluators; Alyx can draft them |
| Evidence on each finding | The supporting messages, linked from the rule | The score and the judge’s reasoning on the observation | Eval label, score, and explanation on the span, trace, or session |
| Finds failure modes you didn’t write a rule for | Suggests new intents from traffic and summarizes newly seen intents in Slack | Not built in; Pulse highlights count, cost, and latency spikes | Signal reviews production traces and groups recurring failures into ranked issues |
| Alerts | Email or Slack, on intent or violation counts, or a custom condition | Slack, webhooks, or GitHub Actions, on observation metrics or score rates | Custom monitors and custom metrics |
| From failure to pull request | Auto Improve: a fired alert starts a run that reads your repository and opens a pull request | Not built in; an alert can trigger your own GitHub Actions workflow | A managed agent with repository access proposes a patch and opens a pull request |
| Pull requests available on | Every plan, including Free | Not applicable | Enterprise only |
| Checks the fix after merge | Re-checks the triggering alert about an hour after merge: improved, unchanged, or regressed | No | Not described in the docs we reviewed |
| Open source | No | Yes, MIT (except ee folders) |
Phoenix is source-available under Elastic License 2.0; AX is not |
| Self-hosting | Enterprise (VPC) | Free | Phoenix free; AX on Enterprise |
| Free tier | 1,000 events a month, 7-day retention | 50,000 units a month, 30 days | 25,000 spans a month, 1 GB, 15 days, 10 Signal issues |
| First paid plan | $49/mo for 10,000 events | $29/mo for 100,000 units | $50/mo for 50,000 spans and 25 Signal issues |
Units, spans, and events are counted differently. Price each tool against one representative month of your own traffic.
The same silent failure in each tool
Take a support agent whose refund tool returns pending, and the agent tells the customer, “Your refund is complete.” Every span ends with status OK. Here is how each tool catches it and what happens next.
Langfuse. Create an LLM-as-a-judge evaluator that maps the final generation’s input, output, and tool calls and returns a boolean such as false_completion. Attach it to a rule that samples production observations. Then create an alert on the share of true scores, routed to Slack or a GitHub Actions workflow. If the evidence lives in several observations (the tool span and the later answer), you need to put both in the observation the judge reads, or score the trace yourself through the API: trace-level evaluators stop producing results on Langfuse Cloud after November 16, 2026. Anything after the alert is yours to build.
Arize AX. Create an online eval task at trace or session granularity with an LLM judge for false completion, and set a sampling rate. Results attach to each trace or session as trace_eval.<name> or session_eval.<name>. If you turn on Signal for the project, it can surface the pattern as a ranked issue with evidence, a likely cause, and a proposed fix, within your plan’s monthly issue limit. On Enterprise, a managed agent with your repository connected can carry that into a pull request.
Agnost AI. Write the rule: “Describe a refund as complete only after the refund tool reports a completed state.” Agnost AI checks each analyzed conversation against it and links the messages that break it. Choose Alert on the rule to fire on its count over a lookback window. When the alert fires, Auto Improve reads the evidence and your linked repository, and opens a pull request with the root cause, the change, the expected impact, and anything that needs human review. About an hour after you merge, it re-runs the alert condition and records whether the fix worked.
For the detectors behind this example and how to turn them into metrics, see how to catch wrong answers that don’t show up as errors in traces.
Where Langfuse is stronger
- Open source and free to self-host. If quality data must stay on your infrastructure at no license cost, Langfuse is the only one of the three that is MIT-licensed. Phoenix is free to self-host but source-available, and Agnost AI self-hosting is Enterprise-only.
- Precise, cheap judges on single calls. Observation-level evaluators are the right tool when a failure is visible in one generation: a hallucinated field, an unsafe output, a wrong format. They run on the observations you filter to, at the sample rate you choose.
- Alert plumbing you control. Score-rate alerts with warning and alert thresholds, no-data handling, and GitHub Actions or webhook destinations make it easy to wire detection into a pipeline you already own.
Where Arize AX is stronger
- Unsupervised issue discovery. Signal reviews production traces and groups failures you never wrote an evaluator for into ranked issues. Agnost AI surfaces new intents, but does not cluster failures without a rule.
- Choice of granularity. One eval task can judge spans, traces, or sessions, with multi-span filters such as “retrieval happened, then an LLM answered.” That is the most flexible eval scoping of the three.
- Evals and experiments in one place. Datasets, experiments, playground replay of traced calls, and prompt optimization sit next to production evals, so a failing session can become a test case without leaving the product.
Where Agnost AI is stronger
- Failures judged across the whole conversation. Rules are checked against everything that happened under one conversation ID, which is where silent failures usually live: a claim in turn five that contradicts a tool result in turn three.
- Rules anyone can write. Product, support, and compliance owners write violations as sentences and read the evidence, without building a judge prompt, a mapping, or a sampling rule.
- A pull request on every plan. Auto Improve is included from the Free plan up. A fired alert is the trigger, so you decide what gets fixed by deciding what you alert on. It connects through a GitHub App, never merges, and leaves secrets, auth, migrations, and CI changes for a human.
- Checks whether the fix worked. After merge, Agnost AI re-runs the condition that triggered the run and marks the change improved, unchanged, or regressed, so a merged pull request is not mistaken for a fixed problem.
Using them together
All three accept OpenTelemetry, so you don’t have to pick one. A common split is Langfuse or Arize for tracing, evals, and experiments, and Agnost AI for conversation-level rules, alerts, and pull requests. Add Agnost AI as a second span processor on your existing tracer provider and set the same conversation ID on every span; the Agnost AI vs Langfuse guide has the code. Remember that each destination counts every span it receives against its plan.
How to choose
| If your team… | Start with |
|---|---|
| Needs open-source, self-hosted quality checks at no license cost | Langfuse |
| Mostly sees failures inside single model calls and wants cheap per-call judges | Langfuse |
| Wants an agent to find failure patterns it was never told to look for | Arize AX |
| Runs evals, datasets, and experiments as one workflow before each release | Arize AX |
| Sees failures that only show up across a whole conversation | Agnost AI |
| Wants non-engineers to define what a failure is, in plain language | Agnost AI |
| Wants detected failures turned into pull requests without an Enterprise contract | Agnost AI |
| Wants deep debugging plus conversation-level rules and fixes | Langfuse or Arize, plus Agnost AI on the same OpenTelemetry data |
Related reading
- How to catch wrong answers that don’t show up as errors in traces
- Agnost AI vs Langfuse: which should you use for your AI agent?
- LangSmith vs Langfuse vs Braintrust vs Phoenix vs Agnost
- Best online evaluation tools for AI agents (2026)
- How to define AI agent failures in natural language
- How to calibrate an LLM judge for agent monitoring
Frequently asked questions
Which tool catches agent quality failures best: Agnost AI, Langfuse, or Arize?
It depends on where the failure shows up. If it is visible in one model call, all three can judge it. If it only shows up across a whole multi-turn conversation, Agnost AI and Arize AX judge at that level; Langfuse recommends judging individual observations and is retiring its trace-level evaluators. If you also want the fix drafted as a pull request, Agnost AI includes that on every plan, and Arize offers it on Enterprise.
Can Langfuse open a pull request when it detects a failure?
Not by itself. A Langfuse alert can trigger a GitHub Actions workflow or call a webhook, so you can build your own fix pipeline on top of it. Langfuse does not investigate the failure or write the change.
Does Arize open pull requests for agent failures?
Yes, on Arize AX Enterprise. Signal, which reviews production traces and groups recurring failures into issues with a likely cause and a proposed fix, is on every AX plan with a monthly issue cap. Managed agents with repository access, which can open the pull request, are included only on Enterprise.
How does Agnost AI turn a quality failure into a pull request?
You write the failure as a plain-language rule or pick an intent, then create an alert on it. When the alert fires, Auto Improve reads the evidence and your linked GitHub repository and opens a pull request with the root cause, the change, the expected impact, and anything that needs human review. About an hour after you merge, it re-checks the alert condition and marks the fix improved, unchanged, or regressed.
Which is cheapest for catching quality failures?
For trace volume, Langfuse and Arize start cheaper: Langfuse Core is $29 a month for 100,000 units and Arize AX Pro is $50 a month for 50,000 spans, while Agnost AI Starter is $49 a month for 10,000 events. Langfuse can also be self-hosted for free. Count the work too: judges you write and run yourself, and the fix pipeline you would otherwise build.
Can I run Agnost AI next to Langfuse or Arize?
Yes. All three accept OpenTelemetry, so one tracer provider can export the same spans to two destinations. Put the same conversation ID on every span so a session in one tool matches a conversation in the other.