---
title: "How to Sample Agent Traces Without Hiding Silent Failures"
description: "Combine a representative trace sample with targeted incident retention, and keep selection probabilities visible when estimating production failure rates."
author: "Agnost AI Team"
datePublished: "2026-10-24T00:00:00.000Z"
dateModified: "2026-10-29T00:00:00.000Z"
canonical: "https://agnost.ai/blog/sample-agent-traces-without-hiding-failures/"
tags: ["OpenTelemetry","Agent Monitoring","Failure Detection"]
---

# How to Sample Agent Traces Without Hiding Silent Failures

Combine a representative trace sample with targeted incident retention, and keep selection probabilities visible when estimating production failure rates.

**Use a representative sample to estimate how often failures occur, and targeted retention to investigate known risks.** Do not combine the two into one unqualified failure rate. A dataset intentionally enriched with errors is useful for debugging but does not represent normal production traffic.

Silent failures make this harder because the trace may contain no exception. Keeping every error trace still misses successful executions that produced wrong answers.

## Choose the sampling boundary deliberately

Head sampling decides early, before the entire trace is known. Tail sampling can inspect more of the completed trace before deciding what to retain. [OpenTelemetry's sampling documentation](https://opentelemetry.io/docs/concepts/sampling/) explains these approaches and the additional operating complexity of tail sampling.

For agent monitoring, also distinguish trace retention from evaluation sampling. You might retain operational metadata for every run while evaluating content for a smaller, approved sample. A downstream evaluator cannot inspect content that the upstream pipeline already discarded.

## Keep two datasets with different jobs

| Dataset | Selection | Primary use |
| --- | --- | --- |
| Representative sample | Known random or probability-based selection | Estimate behavior in an eligible population |
| Diagnostic sample | Errors, high latency, new versions, suspicious outcomes | Investigate incidents and collect regression cases |

You can store them together if every record retains its selection reason and probability where applicable. The important requirement is that analysis can distinguish them.

A diagnostic sample based on tool errors will not discover every unsupported answer. Add periodic review of apparently healthy runs. That review can reveal failure types your current selection rules do not know how to find.

## Preserve complete evidence for selected work

Sampling individual events independently can leave a conversation without its relevant tool result. Define whether you need a full trace, a turn with child spans, or a multi-turn conversation. Correlate identities so selected evidence remains interpretable.

Long-running tasks require special attention. A decision window that closes before the final callback may retain the start and lose the outcome. Inspect the collector's behavior under late arrival and capacity pressure, and measure dropped telemetry.

Do not promise that tail sampling guarantees all important failures. It depends on the rules, available evidence, arrival timing, and operational health of the sampling system.

## Show how selection changes the numbers

Imagine 1,000 eligible runs with 20 real failures. A diagnostic dataset retains all 20 failures and 80 healthy runs. Its apparent failure rate is 20%, while the full population rate is 2%.

These illustrative counts demonstrate selection bias, not a production result. Reporting “20% of our agents fail” from the diagnostic dataset would be wrong.

If inclusion probabilities are known and nonzero for the relevant units, an analyst can use an appropriate weighted estimator. If selection is based on undocumented heuristics or missing data, do not invent a corrected percentage. Report the observed count and its limited scope.

## Separate discovery from alerting

Use broad review samples to discover unknown failure types. Use targeted rules for known incidents. When a new pattern appears, label examples, test a detector, and decide whether the impact justifies an alert.

Record the sample definition with the rule version and agent version. A changed selection policy can change the observed failure rate even when the underlying agent stays the same.

In Agnost, remember that analysis can only use the events you send. Review the [OpenTelemetry integration](https://docs.agnost.ai/otel) and the content captured by your framework before relying on a dashboard denominator.

## Check coverage before interpreting improvement

Track ingestion lag, missing child spans, dropped events, and the fraction of selected runs with sufficient evidence. A lower failure count after an exporter outage is not progress.

The [task-success measurement guide](/blog/measure-agent-task-success-rate/) explains how to preserve unknown outcomes. The [regression workflow](/blog/production-failures-regression-tests/) explains how diagnostic examples can become tests without pretending they are a representative estimate of production quality.