llm-as-judgelisted

# LLM-as-Judge — Checks as Evidence, Holistic Verdict, No Scores Core rule in one line: > **Collect evidence with binary Yes/No checks, let the LLM issue one named > holistic verdict, and never aggregate the answers into a score.** ## Why not rubric scores - **Irreproducible.** Numeric scores on the same input drift between runs (3 vs 4 out of 5). LLMs are bad at independent per-dimension scoring — they get pulled by the overall impression, and central-tendency bias compresses scores into a narrow band that straddles any threshold you pick. - **Dilution.** A sum or satisfaction ratio converts one fatal defect ("the referenced file does not exist") into a small deduction. One dominant No must be able to decide the outcome alone. - **Unexplainable.** Nobody can say why something is a 3.5. A No answer to a concrete question explains itself and doubles as the improvement item. Rubrics were invented to structure *human* evaluation, where assessors can score dimensions independently. For an LLM, invert the design: force the checks it tends to skip, and leave the judgment holistic — that is what it is good at. ## The three principles | Principle | Do | Don't | |---|---|---| | ① Binary checks | Decompose criteria into Yes/No questions with 1-line evidence each | "Rate specificity 1–5" | | ② Named holistic verdict | Pick exactly one verdict from a fixed enum, judging the whole | "Total 12 points → pass" | | ③ No aggregation | Enumerate the No answers as the verdict'