Cost per completed task divides the cost of an eligible workload by its confirmed completed tasks. Include failed attempts and retries in the numerator. Otherwise a cheap model that frequently needs recovery can appear more economical than a system that finishes the job.
Keep model spend visible, but do not confuse it with the total cost of an agent workflow.
Define the denominator first
Pick a task with a verifiable terminal condition: an accepted support escalation, a confirmed booking, or a completed export. Define the observation window and how pending tasks are handled.
A user asking three follow-up questions may still represent one task. One agent turn may attempt several tasks. Counting messages as completed tasks distorts the economics before any cost calculation begins.
Use the outcome measurement contract to separate completed, failed, pending, and unknown outcomes. Report the counts with the cost metric rather than publishing a dollar figure with no cohort definition.
Add the costs your architecture actually incurs
| Cost category | Examples | How to account for it |
|---|---|---|
| Model inference | Input/output tokens, cached input, repeated attempts | Use actual provider units and applicable rates |
| External tools | Search, document processing, browser execution | Count billed operations, not just model calls |
| Infrastructure | Workers, queues, storage, telemetry | Allocate consistently and state the method |
| Evaluation | Model judges and replay workloads | Separate production checks from development experiments |
| Human recovery | Support or engineering review | Report measured time or a clearly labeled estimate |
Avoid double counting a bundled service fee and its included underlying usage. Preserve currency, rate version, and billing period so a price change is not mistaken for an engineering improvement.
An illustrative calculation
Suppose both versions process 100 eligible tasks. Version A costs $10 and confirms 80 completions. Version B costs $12 and confirms 96. Both therefore cost $0.125 per confirmed completion before any recovery overhead.
| Version | Eligible tasks | Total cost | Confirmed completions | Cost per completion |
|---|---|---|---|---|
| A | 100 | $10 | 80 | $0.125 |
| B | 100 | $12 | 96 | $0.125 |
These numbers are invented for arithmetic, not Agnost customer results or current model pricing. Version B completes more tasks at the same unit cost in this example, but the decision still depends on quality, latency, and the cost of handling unfinished work.
If five of A’s unfinished tasks complete later, update the outcome window rather than silently changing the denominator. For a fair comparison, allow both versions the same amount of time to finish.
Find savings that preserve outcomes
Start with repeated successful reads, oversized tool payloads, unnecessary model turns, and retries that cannot succeed. These are often easier to reason about than switching the entire application to a cheaper model.
Then test routing or model changes on task-specific cases. A smaller model might handle one narrow workflow well while failing another. Use held-out cases and compare the rate of unsupported completion claims, not only output similarity.
Anthropic’s evaluation guidance recommends explicit success criteria for application evaluation. Here, the economic decision adds a cost measurement to those task criteria; it does not replace them.
Keep development and production spend separate
A one-time replay experiment should not inflate the reported per-task operating cost forever. Conversely, an evaluator that runs on every production request is recurring operating spend. Label each category so the team knows what a change saves.
In Agnost, use tool-call evidence to identify repeated or unsuccessful work, then reconcile costs with provider usage records. Do not assume a trace contains every invoice item or that a missing token field means a free call.
For specific optimizations, start with duplicate-call detection and retry budgets. Measure the result against the same eligible task cohort. A lower bill is valuable when the agent still finishes the work users asked it to do.