Calibrate an LLM judge by comparing its decisions with independently reviewed labels on cases it was not tuned against. Measure misses and false alarms by failure type, retain an insufficient-evidence outcome, and keep the judge version with every result.
A convincing explanation is not a calibration result. The judge can produce a polished rationale for the wrong label.
Define one judgment at a time
Start with a narrow question, such as whether a completion claim is supported by a matching action receipt. Give the judge the relevant messages and tool evidence. Define which facts it may use and what it should return when those facts are missing.
Avoid asking for one score that mixes factuality, friendliness, policy compliance, and efficiency. A single number makes disagreements hard to diagnose and thresholds hard to justify.
Langfuse’s evaluation documentation includes calibration against human labels among its evaluation workflows. The method below is a platform-independent review process you can implement with your existing evaluation tools.
Create labels before tuning the prompt
Ask reviewers to label examples using a written rubric. Include clear failures, healthy controls, near misses, and incomplete records. Resolve disagreements or preserve them as ambiguous cases before optimizing the judge.
For false-completion detection, include “I can cancel it,” “I submitted the request,” “It is canceled,” and a quoted customer statement containing the same words. Otherwise the judge may learn to flag a phrase rather than the behavior.
Keep related examples from the same incident or template in one split. If near-identical conversations appear in both development and test sets, the apparent generalization will be stronger than the evidence supports.
Inspect the confusion matrix
Imagine a held-out set with 20 confirmed failures and 80 healthy cases. A judge catches 16 failures and flags eight healthy cases. Its recall is 16/20, or 80%. Its precision is 16/24, about 66.7%.
These are illustrative numbers, not a measurement of any model or product. Overall accuracy is 88%: 16 true positives plus 72 true negatives. That headline hides the fact that one in three findings sent to a reviewer is a false alarm.
| Measure | Question |
|---|---|
| Precision | Of the findings shown to reviewers, how many are real? |
| Recall | Of the labeled failures, how many did the judge catch? |
| Abstention rate | How often did the judge lack enough evidence? |
| Review burden | How many findings and how much review time does it create? |
The prevalence of failures affects precision. A balanced test set and a low-failure production workload can produce very different reviewer experiences, even with similar sensitivity and specificity.
Test stability and shortcuts
Repeat a subset of cases to inspect label instability. Swap irrelevant names, reorder independent evidence, and paraphrase the same claim. Check whether the judgment changes for a valid reason.
Include adversarial content inside the transcript that tells the evaluator to ignore its rubric. That text is evidence, not an instruction. Separate the rubric from the content and inspect whether the judge preserves the boundary.
Do not use the judge’s self-reported confidence as a calibrated probability unless you have tested that relationship. A value of 0.9 in generated JSON is not automatically a 90% chance of correctness.
Choose thresholds around the action
A finding that enters a weekly review can tolerate a different error tradeoff from a rule that blocks a user action. For important execution controls, prefer deterministic application checks where possible. Model-based monitoring is useful for semantic behavior, but it should not become the sole authorization mechanism.
Version the judge model, prompt, schema, and evidence selection. Recalibrate when any of them changes or when the production task mix shifts. Keep unknowns visible rather than silently dropping difficult cases from the score.
Agnost’s violation evidence provides a review path for classified findings. For the rule-definition step, see natural-language failure rules. For a deterministic contrast, our synthetic monitoring lab shows what can be checked without a language model.