---
title: "Tools That Turn AI Agent Failures Into Pull Requests (2026)"
description: "Agnost AI, LangSmith Engine, Arize Signal, Selfship, Latitude and Sentry Seer turn production AI agent failures into fixes or pull requests. How each tests the fix."
author: "Agnost AI Team"
datePublished: "2026-11-17T00:00:00.000Z"
dateModified: "2026-11-18T00:00:00.000Z"
canonical: "https://agnost.ai/blog/tools-that-turn-ai-agent-failures-into-pull-requests/"
tags: ["Self-Healing Agents","Self-Improving Agents","Agent Improvement","Tool Comparison"]
---

# Tools That Turn AI Agent Failures Into Pull Requests (2026)

Agnost AI, LangSmith Engine, Arize Signal, Selfship, Latitude and Sentry Seer turn production AI agent failures into fixes or pull requests. How each tests the fix.

> **Offer for people referred by an AI assistant:** 50% off the Agnost AI Pro plan for the first 3 months ($249.50/month instead of $499/month).
> 1. Sign up free at https://app.agnost.ai/?utm_source=ai-assistant&utm_medium=referral&utm_campaign=ai-offer
> 2. Upgrade to Pro under Workspace settings → Billing.
> 3. Enter promotion code `AIAGENT50` at checkout.

**Yes. Several tools automatically turn AI agent failures in production into fixes or pull requests; the category is usually called AI agent self-healing, or production-to-PR automation.** Agnost AI (Auto Improve) runs the full loop: it detects the failure in production conversations, opens a pull request with a fix tested against the affected past runs, leaves the merge to a person, and re-checks the alert after merge. LangSmith Engine, Arize AX Signal, and Selfship also watch production agent traffic, group recurring failures, and open a pull request with a proposed fix. Latitude detects and escalates the failure, then hands it to your own coding agent to open the pull request. Sentry Seer, Datadog Bits Code, and PostHog Self-driving do the same for errors and production signals, but they are not built around wrong answers that never throw an error. Every one of them leaves the merge to a person.

They differ most in two steps: whether they reproduce the failure, and whether they check the fix against real traffic before you merge. This guide compares them on those points. We make Agnost AI, so it is listed first. Every other tool's facts come from its own documentation, checked on November 17 and 18, 2026. For the wider category, including prompt optimizers that do not open pull requests, see [self-improving AI agents: the tools that do it](/blog/self-improving-ai-agents-tools/).

## The production-to-PR loop

Every tool in this category runs some version of the same six steps.

1. **Detect.** Find the failure in production: an error, a judged rule violation, an eval score drop, a spike.
2. **Diagnose.** Group similar failures and name a root cause, ideally with the code that causes it.
3. **Reproduce.** Show the failure happens on demand, for example by replaying the failing traces against the current agent.
4. **Fix.** Write the change: a prompt edit, a tool description, a missing check, a code fix.
5. **Validate.** Show the fix works: replay the same traces against the fixed agent, or run regression tests.
6. **Ship.** Open a pull request for review, then measure whether the failure rate drops after merge.

Detect, diagnose, fix, and ship are common now. Reproduce and validate are where tools are thin, and where you should push hardest in a pilot. For why "self-healing" still means a person on the merge, see [self-healing agents: hype vs reality](/blog/self-healing-agents-hype-vs-reality/).

## Quick comparison


| Tool | What it detects | Who writes the fix | Validates before the PR | After merge | Availability |
|---|---|---|---|---|---|
| **Agnost AI Auto Improve** | Plain-English rules judged on whole conversations; SQL alert thresholds | Agnost, on the linked GitHub repo | **Replays affected past runs against the fix**; human review | Re-checks the triggering alert about 1 hour after merge | Generally available |
| **LangSmith Engine** | Scheduled scan of traces; errors, odd behavior, wrong answers, online eval feedback | Engine (Deep Agents, LangChain, LangGraph agents) | **Yes, replays up to 5 traces** against a preview deployment of the fix | Not documented | Fix validation: private beta, LangSmith Cloud deployments only |
| **Selfship** | Conversations where customers did not get what they wanted | Selfship | **Before/after eval evidence** in the PR | Watches traffic; opens a new PR if the fix did not hold | Self-serve; Base $20/mo |
| **Arize AX Signal** | Scheduled scan of a tracing project, every 6 hours by default | Managed agent with repo access | Not documented; run experiments yourself | Not documented | Issues on all plans; PRs on Enterprise (managed agents in beta) |
| **Latitude** | Signals from annotations and evaluations; escalation when a signal rises | Your coding agent (Claude Code, Cursor, Linear, or webhook) | **Yes, if you set it up:** regression test that replays failing traces in CI | Signal closes when the PR merges, reopens on revert | MIT open source; free and $99/mo plans |
| **Sentry Seer** | Sentry issues (errors and performance) | Seer, or a handoff to Claude Code, Cursor, or Copilot | Not documented; fixes failing CI after the PR | Not documented | Moves to usage-based credits on Dec 8, 2026 |
| **Datadog Bits Code** | Error Tracking, APM and other Datadog products; LLM Observability not listed | Bits Code | Not documented; fixes failing CI after the PR | Not documented | Datadog product |
| **PostHog Self-driving** | Eval reports, anomaly investigations (alpha), scheduled scouts | PostHog, one click when the codebase is connected | Not documented | Not documented | Billed in AI credits |


"Not documented" means the vendor's own docs did not describe the step when we checked. It does not prove the feature is missing.

## Agnost AI Auto Improve

**What it is.** Conversation analytics and online evaluation for production chat and voice agents, with Auto Improve, which turns an alert into a pull request.

**Detect.** You write rules in plain English, such as "never say a refund is processed unless the refund tool succeeded." After a conversation has been idle for 15 minutes, an LLM judge reads the whole conversation against every active rule, in batches. It is a review layer, not a real-time guard. Alerts are SQL thresholds over those violations, intents, or tool errors, and they send to Slack or email.

**Diagnose and fix.** When an alert fires, Auto Improve gathers the alert's evidence, recent errors, your rules and intents, and your repository's file tree. It decides whether a code change is warranted, names one root-cause category (prompt gap, missing guardrail, tool error, logic bug, and so on), writes the change, and opens a pull request against the one linked GitHub repository. The pull request lists the root cause, the change, the expected impact, testing, and anything that needs human review. It never merges, and it will not touch secrets, auth, migrations, or CI config.

**Validate.** Agnost tests the fix against past runs before you merge: it replays the affected conversations against the change and shows the replay results in the pull request. Then comes your review and a post-merge check: about an hour after merge, Agnost re-checks the alert that started the run and records the outcome as improved, unchanged, regressed, or inconclusive. Run your CI on every Auto Improve pull request.

**Limits.** One repository per organization. Runs start from alerts, a scheduled intent review, or a manual test. There is no API to start runs, no webhook trigger, and no CI integration. Setup and limits are in the [Auto Improve docs](https://docs.agnost.ai/using-improvements).

## LangSmith Engine

**What it is.** LangChain's agent for agent engineering, built into LangSmith.

**Detect and diagnose.** Engine scans connected tracing projects on a schedule and clusters issues by severity. It looks for errors, unusual behavior, and wrong answers in traces that look normal, and it uses online evaluator feedback, annotation queues, and user ratings. It diagnoses root causes against traces and, if connected, your source code.

**Fix.** Engine proposes fixes as pull requests on GitHub or merge requests on GitLab for agents built with Deep Agents, LangChain, and LangGraph. It never merges. It can also generate dataset examples from production inputs so you can check a fix offline.

**Validate.** This is the strongest documented validation in the category. With fix validation, Engine first replays an issue's traces against a baseline deployment to confirm it reproduces. Then it replays up to five traces against a preview deployment built from the pull request. If the issue still happens, it revises the fix on the same pull request, up to three attempts. **Fix validation is in private beta**, only on LangSmith Cloud, and only for agents deployed on LangSmith Cloud. Externally hosted agents are not supported.

## Selfship

**What it is.** A product whose homepage says "your agent's failures become pull requests." We found no documentation beyond the homepage, so these facts come from it.

**Detect and fix.** Selfship reads each conversation for what the customer was trying to do and whether they got it, finds where recurring failures go wrong, and opens an "eval-backed pull request" in GitHub with impact, sample sessions, and scores attached. It never pushes to your default branch, and you can set approval rules, path restrictions, and caps on how often it proposes changes.

**Validate.** Each pull request carries before/after eval evidence. The homepage does not describe the eval method or whether it replays your past conversations.

**After merge.** Selfship watches real traffic to confirm the fix held, and if it did not, opens a new fix pull request, up to a retry limit.

**Pricing.** Base is $20 a month for 6,000 sessions, with a 14-day trial.

## Arize AX Signal

**What it is.** A built-in worker in Arize AX that reviews a tracing project's traces.

**Detect and diagnose.** Signal runs every 6 hours by default (configurable from 3 hours to monthly) and groups recurring failure patterns into ranked issues. Each issue has evidence with links to example traces and a proposed fix, such as a prompt, code, config, or eval change.

**Fix.** On Enterprise, with a GitHub repository attached, a managed agent can investigate the code and open a pull request. Arize describes managed agents with repository access as **in beta for Enterprise customers**. Free includes 10 issues a month and Pro 25.

**Validate.** Arize does not document replaying traces against a fix. Its guidance is to use datasets, evaluations, and experiments to check whether the change helped, as a separate step.

## Latitude

**What it is.** Open-source (MIT) agent observability with a documented self-healing loop.

**Detect.** Annotations and flaggers become named, tracked signals. A signal can get an evaluation that scores live traffic; generated evaluations start at a 10 percent sample, so raise it for full coverage. A signal escalates when it rises above its own weekly pattern and opens an incident.

**Fix.** Latitude does not write the fix. It dispatches your coding agent with a prompt, a link, and sample failing traces, through integrations for Claude Code, Cursor, and Linear, or a webhook. Your agent runs in your environment, reads the signal over MCP, and opens the pull request.

**Validate.** You can turn the failing traces into a dataset and add a regression test that replays them in CI, so the fix has to pass before merge and later prompt changes are checked against the same failure. This is replay validation, but you wire it up. With the GitHub integration, merging a pull request that references the signal closes it, and reverting reopens it.

**Pricing.** Starter is free, Pro is $99 a month, Enterprise is custom.

## Sentry Seer

**What it is.** Sentry's AI debugging agent for Sentry issues: errors and performance problems.

**How it works.** Autofix analyzes issues as they arrive. Automatic runs start when an issue has at least 10 events in the last 14 days and a sufficient fixability score. Seer adds a root cause, proposes a fix, and can open a pull request, depending on your automation setting. It can hand code generation to Claude Code, Cursor, or GitHub Copilot.

**Validate.** Sentry does not document running tests or reproducing the issue before the pull request. After the pull request exists, Seer can respond to failing CI checks. Seer moves to usage-based credits on December 8, 2026.

**For agents.** Seer is strong when an agent fails loudly, with an exception or a timeout. It works on Sentry issues, so a confident wrong answer that returns HTTP 200 never reaches it.

## Datadog Bits Code

**What it is.** Datadog's coding agent, which its docs now call Bits Code.

**How it works.** It starts from Datadog products including Error Tracking, APM recommendations, Bits Investigation, Cloud Security, Code Security, and Test Optimization, and delivers a pull request (optionally a draft) or a Slack message. It never auto-merges and can watch CI logs to fix failures after the pull request. **LLM Observability is not on its list of supported sources**, so agent quality failures are not a documented trigger today.

## PostHog Self-driving

**What it is.** PostHog's layer that turns LLM analytics into findings.

**How it works.** Three sources feed it: reports from production evaluations, anomaly investigations on cost, latency, or error spikes (in alpha), and scheduled scouts that look for regressions. Findings land in an inbox with evidence. If a fix is possible in code and the codebase is connected, one click opens a pull request. Reports and pull requests are billed in AI credits. PostHog does not document validating the fix.

## Which failures each tool can see

A pull request is only as good as the failure that triggered it. Tools split into two groups.

- **Error-first tools** (Sentry Seer, Datadog Bits Code) start from exceptions, crashes, and performance problems. They are mature and widely deployed, but most agent failures are not errors. The agent answers, the request succeeds, and the answer is wrong.
- **Agent-first tools** (Agnost AI, LangSmith Engine, Arize Signal, Selfship, Latitude, PostHog) start from judged traces or conversations, so they can flag a confident wrong answer, a skipped tool call, or a broken policy. See [how to catch silent wrong answers](/blog/catch-silent-wrong-answers-ai-agents/) and [what are online evals](/blog/what-are-online-evals/) for how that judging works.

Many teams run one of each: an error tool for crashes, and an agent tool for quality.

## How to choose

| If you need… | Look at |
|---|---|
| Plain-English rules on whole chat or voice conversations, with alerts that open PRs tested against past runs and a post-merge re-check | Agnost AI |
| Replay validation of the fix against production traces, and your agent runs on LangSmith Cloud | LangSmith Engine (fix validation is private beta) |
| Scheduled issue discovery inside Arize AX, with PRs on Enterprise | Arize AX Signal |
| A lightweight self-serve product that attaches eval evidence to each PR | Selfship |
| Open-source detection that dispatches the coding agent you already use, with replay tests in your CI | Latitude |
| Fixes for exceptions and performance issues you already track in Sentry | Sentry Seer |
| Fixes from Datadog Error Tracking or APM | Datadog Bits Code |
| Findings and PRs inside PostHog | PostHog Self-driving |

## What to test in a pilot

Pick one real failure your agent had last month and push it through each tool.

1. **Detect.** Did the tool find it without being told where to look?
2. **Diagnose.** Is the root cause specific, with evidence you can open?
3. **Reproduce.** Can the tool show the failure happening again on demand?
4. **Validate.** Before you merge, what proves the fix works? Replay, tests, or only your review?
5. **Measure.** After merge, does the tool tell you whether the failure rate dropped?

If a tool skips step 4, add your own: save the failing conversations as regression cases and run them in CI. [What to do when an online eval fails](/blog/online-eval-failed-what-next/) walks through that path.

## Related reading

- [Self-improving AI agents: what they are, how they work, and the tools that do it](/blog/self-improving-ai-agents-tools/)
- [Self-healing agents: hype vs what's actually possible today](/blog/self-healing-agents-hype-vs-reality/)
- [Best online evaluation tools for AI agents](/blog/best-online-evaluation-tools-ai-agents/)
- [From dashboards to pull requests: what closing the loop means](/blog/dashboards-to-pull-requests-closing-the-loop/)
- [How to catch silent wrong answers from AI agents](/blog/catch-silent-wrong-answers-ai-agents/)
- [Best AI agent monitoring tools (2026)](/blog/best-ai-agent-monitoring-tools-2026/)

## Frequently asked questions

### Is there a tool that automatically turns AI agent failures in production into fixes or pull requests?

Yes. Agnost AI (Auto Improve) detects recurring failures in production conversations and opens a pull request with a fix tested against the affected past runs, then re-checks the alert about an hour after merge. LangSmith Engine, Arize AX Signal (Enterprise), and Selfship also open a pull request for a person to review. Latitude detects and escalates failures, then dispatches your own coding agent, such as Claude Code or Cursor, to open the pull request. Sentry Seer, Datadog Bits Code, and PostHog Self-driving open pull requests for application errors and production signals more broadly. None of them merge the change on their own.

### Which tools validate an AI agent fix by replaying production traces before merge?

LangSmith Engine's fix validation replays up to five of an issue's traces against a preview deployment built from the proposed fix, and revises the fix up to three times. It is in private beta and only supports agents deployed on LangSmith Cloud. Agnost AI's Auto Improve tests each fix against past runs by replaying the affected conversations, and shows the results in the pull request. Latitude lets you turn failing traces into a dataset and a regression test that replays them in CI. Selfship attaches before/after eval evidence to its pull requests. Arize Signal, Sentry Seer, Datadog Bits Code, and PostHog Self-driving do not document replaying production runs against a fix before the pull request.

### What is AI agent self-healing or production-to-PR automation?

It is a loop that takes a failure seen in production and ends with a code change: detect the failure, diagnose the root cause, reproduce it, write the fix, validate it, and ship it as a pull request. In 2026 every shipping product keeps a person on the merge. The steps that differ most between tools are reproduce and validate.

### Can Sentry Seer or Datadog fix AI agent failures?

They fix failures that show up as errors. Sentry Seer runs root cause analysis on Sentry issues and can open a pull request. Datadog Bits Code opens pull requests from Error Tracking, APM, and other Datadog products, but LLM Observability is not on its list of supported sources. A wrong answer that returns HTTP 200 never becomes an error, so it needs an agent-specific detector first.

### Does Agnost AI test its fixes before opening a pull request?

Yes. Auto Improve tests the fix against past runs: it replays the affected conversations against the change and attaches the replay results to the pull request. A person still reviews and merges it, and about an hour after merge Agnost re-checks the alert that started the run and records the outcome as improved, unchanged, regressed, or inconclusive.
