llm-as-judge-scorerlisted

When output is open-ended — a summary, a support answer, tone, helpfulness — you can't score it with exact match, and human grading doesn't scale. An **LLM-as-judge** does, but only if it's built carefully: an uncalibrated judge produces confident, inconsistent scores that quietly corrupt every downstream decision. This skill designs a judge you can actually trust. ## When to use this skill - Grading subjective or open-ended outputs where there's no single correct string. - Replacing slow, inconsistent manual review in an eval loop. - An existing LLM-as-judge gives scores that don't match your own judgment. ## Instructions 1. **Define the rubric explicitly.** State precisely what's being judged and the criteria. Vague instructions ("rate quality 1–10") produce noise; concrete criteria ("deduct if the answer omits the rotation step, hallucinates a flag, or exceeds 3 sentences") produce signal. 2. **Use a discrete scale with anchors.** Prefer a small scale (e.g. pass/fail or 1–5) with a written description of what each level means. Discrete, anchored scales are far more consistent than a bare 1–10. 3. **Provide reference examples.** Include a few scored examples in the judge prompt — especially boundary cases — so the model calibrates to your standard rather than its own. 4. **Control known biases.** LLM judges favor longer answers, their own model family's style, and the first option in a pairwise test. Mitigate: randomize order in pairwise comparisons, instruct length-neu