llm-as-judge-designerlisted

# LLM-as-Judge Designer Criteria in, a calibrated judge out. The calibration is the exemplars: a judge that has never seen a failure will never find one. ## Verification gates (defined first; output is blocked until all pass) - **G1 — Pass AND fail exemplar per criterion:** every criterion — gates and rubric dimensions alike — ships with at least one concrete pass exemplar and one concrete fail exemplar, written as short realistic outputs, each with its verdict and a one-line reason. Rubric fail exemplars must include a mid-grade failure (a 2–3), not just a strawman 1 — judges drift in the middle. Only-pass, only-fail, or adjective anchors ("1 = not empathetic") fail the gate. - **G2 — Anti-drift instructions present:** the prompt forbids grading anything beyond the listed criteria, forbids scoring gates, requires quoted evidence per verdict, and resolves score ties downward with both candidate sentences cited. Calibration cases appear in mixed pass/fail order with an order-independence instruction. - **G3 — Domain-native exemplars:** exemplars come from the criteria's actual domain and read realistic. Imported or cartoonish examples fail — a judge calibrated on strawmen grades strawmen. ## Steps 1. **Take the criteria as given.** This skill doesn't redesign them — a criterion that can't be exemplified (nobody can write what a failure looks like) is returned to the author as unjudgeable, which is a finding, not a workaround. 2. **Write the exemplar set:** per criterion,