---
title: "AI Agent Monitoring Benchmark: What Different Checks Actually Catch"
description: "Reproduce a 17-case synthetic exercise comparing exception checks, call-count rules, and outcome contracts. Includes fixtures, code, results, and limitations."
author: "Agnost AI Team"
datePublished: "2026-10-11T00:00:00.000Z"
dateModified: "2026-10-29T00:00:00.000Z"
canonical: "https://agnost.ai/blog/agent-monitoring-detection-benchmark/"
tags: ["Agent Monitoring","Agent Evals","Benchmarks","Python"]
---

# AI Agent Monitoring Benchmark: What Different Checks Actually Catch

Reproduce a 17-case synthetic exercise comparing exception checks, call-count rules, and outcome contracts. Includes fixtures, code, results, and limitations.

**An exception detector can miss an agent that executes successfully but produces the wrong outcome.** We built a small, reproducible exercise to show the difference between checking execution status, adding a call-count rule, and checking explicit outcome contracts.

The dataset contains **17 hand-authored cases: 10 failures, six healthy controls, and one case with incomplete evidence**. It contains no customer data, live model calls, vendor integrations, or measurements of real-world failure prevalence.

## Download and reproduce the exercise

Save these files into the same directory:

- [benchmark.py](/blog/examples/agent-monitoring-lab/benchmark.py), the standard-library Python runner.
- [fixtures.json](/blog/examples/agent-monitoring-lab/fixtures.json), every input and expected label.
- [results.json](/blog/examples/agent-monitoring-lab/results.json), the recorded output for comparison.

Run with Python 3.9 or later:

```bash
python3 benchmark.py --output my-results.json
```

No API key, account, or package installation is required. The runner reads local fixtures and writes results only to the output path you specify.

## What the three checks do

**Exception only** flags a recorded execution error. It does not interpret answers, business errors, or external state.

**Exception or call count** adds a simple heuristic: flag runs with more than two recorded tool calls. This deliberately shows what a coarse activity threshold can and cannot establish.

**Contract checks** compare explicit fields: duplicate successful reads, claimed completion against a matching receipt, answer days against retrieved days, source version, tenant identity, handoff acceptance, and business errors. These checks receive structured facts supplied in the fixture. They do not extract those facts from natural language.

All three abstain when the fixture says evidence is incomplete. That flag is provided by the exercise; a production system needs its own way to assess coverage.

## Results from the supplied fixtures

| Check | True positives | False positives | Missed failures | Healthy cases passed | Unknown |
| --- | ---: | ---: | ---: | ---: | ---: |
| Exception only | 1 | 0 | 9 | 6 | 1 |
| Exception or call count | 2 | 1 | 8 | 5 | 1 |
| Contract checks | 10 | 0 | 0 | 6 | 1 |

These counts come from running the published script. The contract checks pass all labeled fixtures because this is a small sanity exercise designed around their explicit contracts. **That is not evidence of 100% detection accuracy on production agents.** The fixtures and rules were developed together; there is no independent held-out benchmark.

The coarse call-count rule catches the repeated search but also flags legitimate polling. A retry after a failed first call passes the more specific duplicate-read check because the first call did not produce a usable result.

## What the exercise teaches

The interesting difference is the information available to each check. A refund receipt for the wrong order can look successful in transport logs. It fails an outcome contract because the object identity is wrong. An agent that uses an outdated policy can produce an answer consistent with its retrieved source and still be wrong for the current task.

These failures require evidence beyond an exception flag. Adding a language model judge would not remove that requirement. A judge cannot reliably verify a refund against a ledger it never sees.

We did not run an LLM judge in this exercise. Doing so would require specifying the judge model, prompt, input visibility, repetitions, cost, and comparison with independent human labels. Reporting made-up judge scores would obscure the exact question this exercise answers.

## Extend it without fooling yourself

Add cases from your own application after removing sensitive data. Include healthy examples that resemble each failure. Keep newly labeled cases separate from cases used to tune the detector, and add an unknown category when evidence is insufficient.

Measure precision and recall by failure type. Separate inference cost from telemetry cost and end-to-end response latency. If a detector requires a business-system lookup, include that lookup in the operational assessment.

Agnost's [behavior rules](https://docs.agnost.ai/using-violations) can help review findings in context; the Python checks here are independent examples, not a representation of Agnost's internal detection implementation. Use [production-to-regression testing](/blog/production-failures-regression-tests/) to turn confirmed incidents into a larger, meaningful evaluation set.