Agnost AIAgnost AIBook a demo
Open +
← All articles
· Last updated:

How to Investigate AI Agent Failures From Slack

Use Slack as the investigation surface for production agent failures: receive grounded summaries, inspect evidence, ask follow-ups, and route fixes without losing context.

  • AI Agent Monitoring
  • Slack
  • Agent Failures
  • Production AI

Yes—you can investigate AI agent failures from Slack without making Slack the system of record. The safe pattern is to send a bounded, evidence-backed summary into a connected channel, keep links to the underlying production interactions, and let thread follow-ups query read-only monitoring data. Engineering changes still move through the repository and normal review process.

Slack is useful because it is already where incident triage and product decisions happen. It becomes harmful when a bot posts vague “quality dropped” messages with no denominator, examples or owner.

What a useful agent-failure alert contains

Every message should answer six questions:

  1. What changed? A count and rate over an explicit time window.
  2. Who or what is affected? Agent, intent, tool, rule, release and user cohort.
  3. What is the evidence? A bounded, redacted example and the supporting trace or interaction.
  4. How confident is the classification? Separate observed failure from incomplete telemetry.
  5. Why does it matter? User, revenue, safety or operational impact.
  6. What should happen next? Inspect, assign, create a regression case or tune the alert.

A weak alert says:

Agent quality may have decreased.

A useful alert says:

Refund completion claims exceeded 3% over the last 6 hours: 14 of 382 eligible interactions. Eleven claims have no matching terminal receipt; three have incomplete receipt telemetry. The increase began after release 2026.10.28.2 and is concentrated in refund_order.

The second message gives the team a denominator, scope, uncertainty and an investigation path.

How the Agnost Slack workflow works

Agnost alerts can deliver to a connected Slack channel. Daily summaries combine deterministic volume metrics with bounded analysis of errors and product signals. Follow-up questions in the report thread enter a read-only investigation flow that can query the relevant monitoring data while preserving the report context.

This means Slack is the collaboration surface, while Agnost remains the evidence store. The report does not need to paste full transcripts or unrestricted trace payloads into a channel.

The setup is:

  1. Connect a Slack destination in Agnost.
  2. Enable the daily summary or create a custom alert.
  3. Choose an actionable intent, violation, tool or failure condition.
  4. Set the lookback window and threshold.
  5. Send a test alert and review the rendered evidence.
  6. Adjust the threshold after observing real traffic.

See the Agnost alerts documentation for the current product flow.

Three Slack alerts worth creating first

1. Unsupported completion claims

Condition: the agent claims a high-impact action completed, but the interaction has no matching receipt in the required terminal state.

Include: action type, entity class, eligible interaction count, unsupported count, incomplete-telemetry count and one redacted example.

Do not: include raw payment data or treat missing telemetry as definitive proof that the action failed.

2. Repeated unresolved requests

Condition: a user asks for the same goal again within a conversation after the agent answered.

Include: intent, repeat count, turns before repetition, affected users and the first answer that failed to resolve the request.

Tune: exclude legitimate multi-step flows where repetition is part of confirmation.

3. Tool loops or ignored tool evidence

Condition: an equivalent read-only tool call repeats beyond a justified retry budget, or the final answer contradicts a successful tool result.

Include: tool name, normalized argument fingerprint, call count, tool outcome and relevant answer excerpt.

Tune: distinguish retries after timeouts from repeated successful calls.

A practical investigation thread

Suppose the daily report shows a spike in wrong-policy answers. A useful Slack thread proceeds from aggregate signal to specific evidence:

  1. “Which intent and release account for the increase?”
  2. “Show the bounded distribution by retrieved policy version.”
  3. “Give one redacted example where the retrieved policy and answer disagree.”
  4. “Was the wrong version retrieved, or did the answer contradict correct evidence?”
  5. “How many healthy interactions use the same retrieval path?”
  6. “Create a regression-test brief from the confirmed failure pattern.”

The questions narrow the cause before anyone changes a prompt. If retrieval supplied the wrong policy, rewriting the final-answer prompt may make the response more fluent while preserving the underlying bug.

Keep sensitive data out of Slack

Slack channels often have broader membership and retention than production telemetry systems. Treat every alert as a new disclosure surface.

  • Redact personal information before delivery.
  • Prefer stable internal identifiers over names, emails or raw account data.
  • Include short excerpts, not full conversations.
  • Link authorized reviewers back to the evidence system.
  • Keep tool credentials, hidden prompts and authorization headers out of messages.
  • Match the channel to the sensitivity of the monitored workflow.

For regulated or high-risk workloads, route the alert without the example and require authorized reviewers to open the source interaction.

From Slack finding to a tested fix

Slack should not directly deploy a prompt or code change. The reliable handoff is:

  1. Confirm the failure from production evidence.
  2. Cluster equivalent interactions and select representative cases.
  3. Preserve healthy controls from the same workflow.
  4. Convert the failure into a regression case.
  5. Implement the smallest candidate change.
  6. Replay baseline and candidate versions.
  7. Open a pull request with evidence, test results and limitations.
  8. Review, merge through normal controls and monitor recurrence.

Read Turn Production Agent Failures Into Regression Tests for the dataset contract and From Production Failures to Pull Requests for the review workflow.

When Slack is the wrong surface

Do not use Slack as the primary investigation interface when:

  • the evidence requires long transcripts or dense span trees;
  • access depends on row-level customer permissions;
  • the failure involves secrets, health data or payment details;
  • reviewers need to compare many runs side by side;
  • the decision requires replaying or scoring a candidate change.

In those cases, send only the alert summary and a deep link. Use the dashboard, Claude, Codex or Cursor through Agnost’s OAuth-protected MCP server for deeper read-only investigation, then return the decision and owner to the Slack thread.

Slack investigation checklist

  • Alert has a count, rate and eligible denominator.
  • Time window, agent and release are explicit.
  • Evidence is redacted and bounded.
  • Missing telemetry is labeled separately from observed failure.
  • Message links to the underlying interaction.
  • Alert has an owner and recommended next action.
  • Follow-up access is read-only.
  • Generated fixes require tests and pull-request review.
  • Threshold is reviewed after traffic or agent behavior changes.

The goal is not to move the entire monitoring product into chat. It is to shorten the distance between a credible production signal and the people who can investigate it—without losing evidence, privacy or engineering controls.

Keep reading
Oct 29, 2026AI Agent Monitoring: How to Catch Bad Answers in ProductionA practical guide to monitoring production AI agents for wrong answers, silent failures, bad tool use, and unresolved user goals—not just exceptions and latency.Jul 1, 2026Your Agent Is Failing in Production Even When Your Evals PassPassing evals means your agent handled the cases you expected. It does not mean users are succeeding in production. Here is how to find the gap before it becomes churn.Jul 31, 2026The July Agent Analytics PlaybookA practical playbook for AI agent teams: measure production conversations, catch silent failures, prioritize fixes, and close the loop from user chat to shipped improvement.