judge-calibration-auditorlisted

# Judge Calibration Auditor Disagreement is the signal, not the noise. Each divergence gets diagnosed and corrected — and the correction is never "trust the judge". ## Verification gates (defined first; output is blocked until all pass) - **G1 — Never auto-resolved:** no disagreement is settled by presuming the judge correct — and none by presuming the human correct without analysis. Every disagreement gets a classification and a stated, concrete correction (rewritten anchor text, added exemplar, anti-drift instruction, or an escalation question). "The judge's read is defensible, accept it" fails the gate. - **G2 — Systematic vs. noise separated:** disagreements sharing direction and pattern (n≥2) are systematic and drive anchor rewrites; singletons stay provisional (collect more) unless they show a clear rule-import. Rewriting anchors on one data point fails, as does dismissing a repeated pattern as noise. - **G3 — Honest arithmetic, untouched labels:** agreement stats shown as they are (7/12 is 58%, per-dimension), no invented aggregate quality score, and human labels never edited — a suspect human label becomes a question to the criterion owner, not a correction. ## Steps 1. **Tabulate:** per case — human label, judge verdict, direction and size of gap, the output's relevant character (one line). Compute agreement overall and per criterion. 2. **Cluster the disagreements** by direction + pattern. Two same-direction gaps on the same output pattern are one systematic fi