--- title: "What LLM-as-a-Judge Costs in Production, and How to Cut It" description: "Online evals with an LLM judge cost judge tokens plus platform fees, and both scale with traffic. Here is the formula, a worked example, and the levers that cut cost without hiding failures: filtering, sampling, smaller judges, batching, and code checks." author: "Agnost AI Team" datePublished: "2026-11-07T00:00:00.000Z" dateModified: "2026-11-07T00:00:00.000Z" canonical: "https://agnost.ai/blog/llm-as-a-judge-cost-production/" tags: ["Online Evals","LLM as a Judge","Agent Evals","AI Agent Cost"] --- # What LLM-as-a-Judge Costs in Production, and How to Cut It Online evals with an LLM judge cost judge tokens plus platform fees, and both scale with traffic. Here is the formula, a worked example, and the levers that cut cost without hiding failures: filtering, sampling, smaller judges, batching, and code checks. **The cost of LLM-as-a-judge is judge tokens plus platform fees, and both grow with traffic.** A rubric that costs a fraction of a cent per conversation is invisible in a pilot and a real line item at a million conversations a month. The good news is that most of the cost is avoidable without hiding failures, if you choose what to judge before you choose how. For the setup around this, read [how to set up online evals in production](/blog/how-to-set-up-online-evals-production/). ## The formula ```text monthly judge cost = judged units per month × judge calls per unit × (input tokens per call × input price + output tokens per call × output price) + platform charges (per score, event, span, or retained trace) ``` Each term is a lever: - **Judged units** depend on traffic, the unit (span, trace, or conversation), filters, and sampling. - **Judge calls per unit** depend on whether each evaluator runs separately or several rules share one call. - **Input tokens** are the transcript, the rubric, and any retrieved evidence. They usually dominate. - **Output tokens** are small for pass/fail verdicts and large if you ask for reasoning on every case. - **Platform charges** vary by vendor. Braintrust bills per score, PostHog counts each evaluation as an event, and LangSmith moves evaluated traces to extended retention. ## A worked example Assume a support agent with these numbers. Replace them with yours, and use your provider's current prices. | Input | Value | | --- | --- | | Conversations per month | 200,000 | | Average turns per conversation | 8 | | Transcript tokens per conversation | 3,000 | | Rubric tokens per evaluator | 400 | | Evaluators | 5 | | Output tokens per verdict | 50 | | Judge price (placeholder) | $P_in per million input tokens, $P_out per million output | **Option A: five span-level evaluators on every model call.** With 8 calls per conversation, that is 200,000 × 8 × 5 = 8 million judge calls a month. Even with short inputs per call, this is the most expensive design, and it still cannot see multi-turn failures. **Option B: five separate conversation-level evaluators.** 200,000 × 5 = 1 million calls, each reading about 3,400 input tokens: 3.4 billion input tokens a month. **Option C: one conversation-level call checking all five rules.** 200,000 calls, each reading about 3,000 + 5 × 400 = 5,000 input tokens: 1 billion input tokens a month, under a third of option B. **Option D: option C, filtered.** If only 40% of conversations touch the topics the rules cover, that is 80,000 calls and 400 million input tokens. Same coverage of the failures that matter; roughly an eighth of option B's tokens. ## The levers, in order of impact ### 1. Filter before you sample A refund rubric does not need to read password-reset conversations. Filter by intent, tool used, agent, or customer tier first. Filtering keeps full coverage of the relevant slice; sampling throws away coverage everywhere. ### 2. Share the transcript across rules The transcript is the expensive part. Checking several related rules in one call reads it once. The trade-off is that one combined prompt is harder to calibrate, so keep the number of rules per call reasonable and test the combined judge against labels. ### 3. Move deterministic checks to code Schema validation, error codes, latency, duplicate tool calls, and forbidden strings do not need a model. Each one moved to code removes a judge call per unit. ### 4. Use batch processing when latency does not matter Online evals are retrospective by design. Most providers price batch requests lower than synchronous ones, in exchange for slower turnaround. For a review layer, a delay of minutes or hours rarely matters. ### 5. Test a smaller judge Label 50 to 100 conversations, then run your strong judge and a cheaper one. Compare both to the labels. If the cheaper one misses severe failures, keep the strong judge for those rules and use the cheaper one for the rest. [How to calibrate an LLM judge](/blog/calibrate-llm-judge-agent-monitoring/) covers the comparison. ### 6. Ask for less output Reasoning on every verdict multiplies output tokens. Ask for a verdict and a pointer to the failing message. Request an explanation only on failures, or generate it later when someone opens the case. ### 7. Sample, carefully Sampling is right for common, low-severity behavior such as tone or verbosity. It is wrong for rare, severe failures. If a failure happens 20 times a month and you sample 5%, you expect to catch one, and in about a third of months you catch none. [Sampling agent traces without hiding failures](/blog/sample-agent-traces-without-hiding-failures/) covers stratified approaches. ## Hidden costs to include - **Platform fees on evaluation output.** Scores, events, or retention upgrades can cost more than the tokens. - **Re-judging.** Every rubric change you want to backfill re-reads the history. - **Human review.** Calibration labels and alert triage take engineer time, which usually outweighs the token bill. - **Truncation.** Long conversations may exceed the judge's context, and truncating them changes what the judge can see. ## How Agnost AI handles judge cost Agnost runs its judge on its own infrastructure, so there is no provider key to connect and no separate judge bill. Usage is metered by events on your plan. The pipeline uses three of the levers above by default: one judge call checks a conversation against all active rules at once, conversations are judged once after they have been idle for 15 minutes rather than turn by turn, and requests go through a batch API. There is no sampling, so every analyzed conversation is judged up to your plan's monthly event limit. The [online evaluation tools comparison](/blog/best-online-evaluation-tools-ai-agents/) lists how other vendors price evaluation, and the [AI agent monitoring pricing guide](/blog/ai-agent-monitoring-pricing-cost-guide/) covers the rest of the monitoring bill. ## Frequently asked questions ### How much does LLM-as-a-judge cost? Judge cost per month is roughly: judged units × evaluators per unit × (input tokens × input price + output tokens × output price). Add any platform charge per score, event, or span. Conversation-level judges read more tokens per judgment but need fewer judgments than span-level ones. ### How can I reduce LLM judge costs? Filter to the conversations an evaluator applies to, combine related rules into one judge call over the same transcript, use batch APIs where latency is not critical, move deterministic checks to code, test whether a smaller judge agrees with your labels, and sample only common, low-severity failures. ### Should I sample production traffic for LLM-as-a-judge? Sample common failures where a small sample still finds plenty of examples. Do not sample rare, severe failures: at a 5% sample, a failure that happens 20 times a month is missed entirely in about a third of months. Filtering to the relevant slice is usually a better saving than sampling. ### Is a cheaper judge model good enough? Sometimes. Label 50 to 100 conversations, run both the strong and the cheap judge on them, and compare each against the human labels. Use the cheap judge only where its false negatives on severe failures are acceptable, and re-test whenever you change the rubric.