Production traces are more than debugging records. Once an agent’s job becomes predictable, those traces can become the evidence, evaluation set, and training material for a model specialized to that workflow.
We tested that idea on a narrow identifier-extraction workload. Agnost trained a specialized model from historical production traces, then compared it with Claude Opus 4.8 on 780 held-out customer traces.
Results
| Metric | Claude Opus 4.8 | Agnost specialized model | Change |
|---|---|---|---|
| Task success | 71.5% | 87.9% | 22.9% higher |
| Median latency | 4.8 seconds | Under 0.5 seconds | 90.2% lower |
| Tail latency | 11.2 seconds | 1.35 seconds | 87.9% lower |
| Cost per 1,000 completed tasks | $42.00 | $2.30 | 94.5% lower |
The task-success change is relative: a 16.4 percentage-point absolute increase from 71.5% to 87.9%.
What we evaluated
- Workload: identifier extraction for a production agent.
- Evaluation unit: one complete customer trace, including the information available to the agent and its resulting task outcome.
- Evaluation set: 780 customer traces held out from model training.
- Baseline: Claude Opus 4.8 running the same workload.
- Measures: task success, median latency, tail latency, and inference cost per 1,000 completed tasks.
The model was optimized for this workload, not for open-ended general reasoning. Frontier models remain the right choice when tasks are novel, broad, or difficult to constrain.
Why production traces matter
Generic training data teaches a model to handle many domains. A production agent usually repeats a much smaller set of jobs: route a request, select a tool, retrieve a record, extract identifiers, or format a response.
Agnost starts by monitoring the production agent and finding the silent failures that ordinary error monitoring misses. Those failures become explicit evaluation cases. Candidate fixes and models are tested against held-out traces before they are promoted.
This closes a loop that otherwise stays manual:
- Monitor real agent behavior.
- Find recurring failures and successful patterns.
- Turn traces into evaluation and training sets.
- Train a workload-specific model or test a smaller change.
- Evaluate the candidate on held-out traces.
- Deploy only after the measured result improves.
For prompt and tool changes, see how to turn production failures into regression tests. For model selection, read how to measure task success without inflating it.
Scope and limitations
This is a first-party benchmark from one customer workload. It is not evidence that one specialized model will beat Claude Opus 4.8 on unrelated tasks, and it is not an independently replicated general-purpose model benchmark.
The raw traces cannot be published because they contain customer production data. We are publishing the sample size, workload, comparator, metrics, raw aggregate values, and limitations so teams can judge the result in the correct scope. The original launch results were also published in Parth Ajmera’s announcement and independently summarized by Founderland.
The practical takeaway
Do not replace a frontier model because a smaller model is cheaper. Replace it only when a held-out evaluation set proves that a specialist is more reliable for the actual production workload.
Agnost monitors production AI agents, finds silent failures, and turns trace data into tested fixes and specialized models. If your agent repeats a narrow workflow at production volume, talk to the founders about building the evaluation set before choosing the model.