eval-judgelisted
Install: claude install-skill ychampion/ystack
# Eval judge
A grader that doesn't match the user's judgment makes every later number a lie. This skill builds
one, then proves it agrees with the user before anyone trusts its scores.
## Before you start
- **Cost.** Every grader run calls a model once per output. Estimate runs x outputs x tokens and tell
the user. Use a model the user is already signed in to (a CLI subscription, a local model).
**Never use a paid API key without the user's explicit okay for this task**, and agree a spend cap.
- **Code first.** If a property can be checked by code (valid JSON, tests pass, length, a required
field, a banned word), check it with code. Use a model grader only for judgment calls.
## Procedure
### 1. Collect real outputs
Gather about 20 real outputs of the thing being judged, with their inputs. Real runs beat invented
ones. Include clear passes, clear failures and borderline cases. Save them as one file per case or one
JSONL file: `{id, input, output}`.
### 2. Split off a held-out set
Put about a quarter aside, untouched until step 7. Tuning the rubric on them makes the final check
worthless.
### 3. Get the user's labels
Show the user each working-set case and ask for pass/fail plus one line of why. The why is the raw
material for the rubric. Don't label for them; if the user can't decide, mark it unclear and leave it
out.
### 4. Write the rubric
From the user's reasons, write 3-6 criteria. Each one:
```
<criterion>: pass if <observable condition>; fail if <obse