← ClaudeAtlas

ai-eval-suite-builderlisted

Builds LLM evaluation harnesses: regression datasets, scoring rubrics, prompt drift detection and benchmarks. Use when replacing eyeballed output quality with automated evals.
poorvith-mp/skills-developer · ★ 0 · AI & Automation · score 65
Install: claude install-skill poorvith-mp/skills-developer
# AI Evaluation Suite Builder Architects automated evaluation infrastructure for AI/LLM systems to ensure output quality, detect drift, and prevent regression. ## Phased Workflow ### Phase 1: Evaluation Dataset Design 1. Curate golden test datasets with human-annotated ground truth across representative task categories. 2. Define scoring dimensions: Accuracy, Relevance, Completeness, Tone Adherence, Safety, Latency. 3. Build adversarial test cases targeting known failure modes (hallucination, refusal, format violation). ### Phase 2: Scoring & Comparison Infrastructure 1. Implement automated scorers: exact match, semantic similarity (embedding cosine), LLM-as-judge, regex validators. 2. Build A/B comparison pipelines for prompt version testing (old prompt vs new prompt). 3. Set up drift detection: track score distributions over time windows and alert on statistical deviation. ### Phase 3: CI Integration & Reporting 1. Integrate eval suite into CI/CD pipeline as a required gate before prompt/model deployment. 2. Generate eval reports with pass/fail rates, score distributions, and regression flags. 3. Maintain versioned eval datasets alongside prompt versions for reproducibility. ## Verification & Quality Checklist - [ ] Golden dataset covers minimum 50 representative test cases per task category. - [ ] Scoring rubrics are deterministic and reproducible across runs. - [ ] Drift detection thresholds calibrated against baseline performance windows. - [ ] Eval pipeline runs i