evaluating-ai-outputlisted

# Evaluating AI output Code is deterministic; LLM output isn't. A feature that "worked once" can fail the next call on the same input. **Evals are the unit tests of AI work** — they measure how *often* and how *well* the output meets the bar, not just that it can. ## Read first (always) List `learnings/` and read anything relevant — provider quirks, rubric calibration, and known-flaky cases for this project belong there. ## Define expected behavior BEFORE you implement Write the eval first: the inputs, what a good output looks like, and what must never happen. If you can't state how you'd grade it, you don't yet understand the feature. ## Measure across repeated trials Run each case **k times** (LLM output varies) and report: - **pass@k** — *at least one* of k attempts succeeds. Measures **capability** ("can it do this at all?"). Typical target: pass@3 > 90%. - **pass^k** — *all* k attempts succeed. Measures **stability/reliability** ("does it do this every time?"). Use for **critical paths** (auth, money, irreversible actions). pass^3 means 3 consecutive clean runs. A feature can have high pass@k but low pass^k — impressive once, unreliable in production. Match the metric to the risk. ## Three graders (use the cheapest that's trustworthy) | Grader | How | Use for | |---|---|---| | **Code-based** | Deterministic check — regex/`grep`, schema/JSON validation, an assertion, a tool call that must appear | Anything machine-verifiable (format, presence, exact values). Al