← ClaudeAtlas

eval-judgelisted

Build a grader for an agent's or model's outputs that agrees with the user's own judgment: collect about 20 real runs, have the user label them, write a rubric, run the grader, measure agreement, fix the disagreements, repeat, and check on a held-out set. Works with any model or CLI. Use for "build an eval", "LLM as judge", "grade these outputs", "auto-score the agent", "is the new prompt better", or before tuning a prompt or skill by numbers.
ychampion/ystack · ★ 4 · AI & Automation · score 77
Install: claude install-skill ychampion/ystack
# Eval judge A grader that doesn't match the user's judgment makes every later number a lie. This skill builds one, then proves it agrees with the user before anyone trusts its scores. ## Before you start - **Cost.** Every grader run calls a model once per output. Estimate runs x outputs x tokens and tell the user. Use a model the user is already signed in to (a CLI subscription, a local model). **Never use a paid API key without the user's explicit okay for this task**, and agree a spend cap. - **Code first.** If a property can be checked by code (valid JSON, tests pass, length, a required field, a banned word), check it with code. Use a model grader only for judgment calls. ## Procedure ### 1. Collect real outputs Gather about 20 real outputs of the thing being judged, with their inputs. Real runs beat invented ones. Include clear passes, clear failures and borderline cases. Save them as one file per case or one JSONL file: `{id, input, output}`. ### 2. Split off a held-out set Put about a quarter aside, untouched until step 7. Tuning the rubric on them makes the final check worthless. ### 3. Get the user's labels Show the user each working-set case and ask for pass/fail plus one line of why. The why is the raw material for the rubric. Don't label for them; if the user can't decide, mark it unclear and leave it out. ### 4. Write the rubric From the user's reasons, write 3-6 criteria. Each one: ``` <criterion>: pass if <observable condition>; fail if <obse