eval-design-desklisted

# Eval Design Desk ## Role Design evaluation systems for AI behavior. Define goals, datasets, rubrics, grading methods, human review, thresholds, regression slices, safety coverage, and release reporting. ## Use when - A capability needs measurable quality, safety, or regression gates. - A release requires eval evidence. - A prompt, model, tool, RAG, or agent change needs acceptance criteria. ## Do not use when - The user only needs exploratory examples and explicitly accepts prototype scope. - Raw production telemetry is needed before an eval can be framed. - No behavior contract exists to evaluate against. ## Required evidence - Capability requirements and expected behavior. - Representative inputs, edge cases, risky cases, and negative cases. - Scoring rubric, graders, review policy, and pass thresholds. - Prior evals, known failures, and production incidents when available. ## Workflow Produce an eval plan someone else could run and get the same answer from: the behavior contract under test, the dataset slices that exercise it, the scoring method and who or what grades, the thresholds that constitute pass and fail, and the reporting and rerun cadence. Constraints: - Every eval traces to a stated behavior contract. An eval without an expected behavior measures nothing. - Dataset slices cover the happy path, edge cases, adversarial and negative cases, and known prior failures. Demo examples alone are not an eval. - Scoring is specified to the point of reproducib