eval-authoringlisted

# Eval Authoring ## Purpose Give LLM apps a repeatable way to escape "vibes-based" development, where a prompt is changed, a few outputs are eyeballed, and the change ships with no evidence it helped. An eval suite turns that into a measured, versioned, CI-gated check: every prompt or model change is scored against fixed cases, and a regression fails the build. This skill standardizes what to test, how to grade it, and how to wire it in, riding an existing framework (promptfoo, DeepEval, Ragas) when one is present, and falling back to plain unit tests plus an LLM judge when none is. ## The three grader types | Grader | Use for | Cost / stability | Example | |---|---|---|---| | **Assertion** | Deterministic, checkable facts about the output | Cheap, stable, run every commit | `is-json`, `json-schema`, `contains`, `regex`, `not-contains` (banned strings), latency/cost budget | | **LLM-as-judge** | Qualities assertions can't capture | Costs a call, needs a threshold | Faithfulness, helpfulness, tone, "answers the question", pairwise vs. baseline | | **Golden dataset** | Regression over a curated set of real cases | Grows from production traffic | `input → expected/assert`, tagged, versioned in the repo | Prefer assertions wherever the property is checkable, they are free and non-flaky. Reach for a judge only for genuinely subjective qualities, and always give it a pass threshold. Use the golden set to lock in behavior you've already fixed so it never regresses. ## Authoring