← ClaudeAtlas

eval-foundationslisted

The principles behind every LLM judgment and eval, for designing or revising one.
shimo4228/claude-harness · ★ 4 · AI & Automation · score 83
Install: claude install-skill shimo4228/claude-harness
# Eval Foundations — Principles Behind Every Judgment Read this skill when you design a judgment or revise one: a judge, an eval, a review gate, a reader loop. A run of an existing judgment needs only the skill that runs it; that skill holds everything the run requires. ## The principles 1. **Reach the verdict by holistic judgment, not by summing rubric points.** A measure that becomes the target breaks down (Goodhart). Holistic judgment is what to aim for. 2. **Treat noise and bias separately.** Repeating the same judge reduces only the wobble of the moment (occasion noise). The judge's stable habits, and a shift that lands on every judge in the same direction, survive repetition. 3. **Mix judges by lineage rather than by number** (for example, Claude and Codex). Keep a disagreement between lineages as a split instead of averaging it. Habits that all lineages share still survive the mix; only outside feedback shows them. 4. **A judgment that guards the floor may be a panel; a judgment that picks the ceiling stays with one eye.** Guarding the floor means finding gaps or a dominant No. Picking the ceiling means choosing which one is best. 5. **The overall judgment and the per-criterion evaluation bleed into each other.** Decide in advance which one comes first and what each judge is shown. 6. **Keep the loop that fixes the work apart from the judgment that evaluates it.** A judgment inside the loop is learned by the loop and pushed on, so pla