build-eval-datasetlisted

# Build an eval dataset Evals are only as good as the dataset behind them. A metric run over toy inputs gives you confident, wrong signal. This is how to build a set that actually reflects your app and catches real regressions. ## What makes a dataset good - **Representative** of real usage, not made-up easy cases. The distribution should match production. - **Covers the hard cases** you already know about: edge cases, ambiguous inputs, adversarial ones, the bug reports. - **Has a clear success definition** per item: a reference answer, or a rubric a judge can apply consistently. - **Small enough to iterate, big enough to trust.** Start at 20-50 items; grow toward a few hundred as the app matures. You do not need thousands to start. - **Versioned** in the repo next to the code it tests. ## Where to get the data (best to worst) 1. **Real production traffic** (best). Pull real inputs from your traces (this is the payoff of having observability). Sample across the distribution, and deliberately include failures you saw. 2. **Beta / internal usage.** Dogfood inputs before you have prod traffic. 3. **Domain experts writing cases.** For regulated/specialized apps, have an expert write inputs + gold answers. 4. **LLM-generated cases** (last resort, use with care). Have a model generate candidate inputs, then a human curates. Never ship purely synthetic gold answers unverified. ## Build it in layers 1. **Smoke set (5-10):** obvious cases that must always pass. Run these on eve