Judging answers without a ground truth

Benchmarks need a right answer

Evaluation as an industry is built on tasks with keys. Multiple choice, extraction, translation against a reference, code that either passes the test or does not. These are real tasks and the measurements mean something.

They are also not what most production systems do. Summarise this thread. Draft a reply. Explain this clause to someone who is not a lawyer. Decide whether this ticket needs a human. None of these have a correct string, and several have a range of good answers wide enough that two competent people would disagree about the ordering.

What we found ourselves doing

Grading on taste, and calling it accuracy. Someone reads forty outputs, forms an impression, and that impression becomes a number in a document. The number then gets compared week to week as though it were measuring the same thing, which it is not, because the person is different, the mood is different, and the forty samples are different.

This is not a criticism of the people doing it. It is what you do when the alternative is measuring nothing. But it is worth being honest that it is a vibe with a decimal point.

Rubrics move the problem

The standard next step is a rubric: break the judgement into dimensions — faithfulness, completeness, tone, whether it invented anything — and score each. This helps, genuinely. Disagreement between reviewers drops, and the dimensions make failure legible in a way a single score never does.

What it does not do is remove the judgement. It relocates it into the rubric design, where it is at least visible and arguable. Whether “completeness” is worth as much as “faithfulness” is a product decision wearing a lab coat.

Measuring the judge

Using a model as the judge is now routine, and it works well enough to be dangerous. A judge model is fast, consistent, and will happily reproduce its own preferences at scale — including a documented fondness for longer answers, for its own outputs, and for confident phrasing regardless of whether the confidence is warranted.

So the harness needs a harness. We are keeping a small, slow, human-graded set as the calibration reference, and treating agreement between the model judge and that set as the thing under test — not the model judge’s scores themselves. When agreement drifts, the judge is the suspect, not the system.

If you cannot measure your judge, you are not measuring your system.

The uncomfortable implication is that evaluation has a fixed human cost that does not go to zero, no matter how good the automation gets. It can be made small. It cannot be removed.