llm-judge-auditorlisted

# Auditing the judge An LLM judge is a model with an eval of its own that nobody ran. Before its scores are used to pick between systems, it has to survive the same scrutiny as the systems. ## The refusal **No human agreement number, no judge scores.** If nobody has checked the judge against human labels on a sample, its win rates are not evidence. Say that and stop. ## The four checks 1. **Self-preference.** A judge from the same family as one of the candidates favours it. Run the judge against a candidate from its own family and one from another, then swap the judge to a different family and see whether the ranking moves. If it moves, report both and do not pick the flattering one. 2. **Position bias.** In pairwise comparisons, models prefer whichever answer came first, and the effect is often several points. Shuffle presentation order per item and measure it. Report the size of the bias as its own number, not as a footnote. 3. **Human agreement.** Sample at least 100 judged items and have humans label them. Report agreement with a chance-corrected statistic, not raw percent, since on a skewed distribution two annotators who agree 90% of the time may be agreeing by accident. Below moderate agreement, the judge is not measuring the thing. 4. **Verbosity and format preference.** Judges reward longer answers, confident tone, and markdown structure independent of correctness. Test by padding a correct short answer and seeing whether the sco