eval-driven-developmentlisted

# Eval-driven development (TDD for LLMs) Changing a prompt and eyeballing one output is how LLM apps quietly regress. Eval-driven development flips it: define what "good" means as a small test set, then iterate until you pass it. It is the single habit that separates apps that get more reliable over time from ones that drift. ## The loop 1. **Write the eval first.** Before touching the prompt, collect 10 to 30 real input cases and define what a good output looks like (a reference answer, or a rubric for LLM-as-a-judge). Include the failure cases you already know about. 2. **Run it against the current version.** This is your baseline score. Now you have a number, not a feeling. 3. **Make one change.** New prompt, different model, changed retrieval. One at a time so you know what moved the score. 4. **Re-run the evals.** Kept the score or improved it? Keep the change. Dropped it? Revert. No debate. 5. **Add every new bug as a case.** When something breaks in production, capture that input as a new eval case before you fix it. The suite grows into a regression net. ## Wire it into the workflow - Keep the eval set in the repo (versioned), next to the prompts it tests. - Run it in **CI on every PR** that touches prompts, models, or retrieval, with thresholds that fail the build on a regression. - Pull real cases from **production traces** (this is where observability feeds evals) so the suite reflects reality, not toy inputs. See the `add-llm-evals` skill for framework setup