agent-evaluation-framework-builder

# Agent Evaluation Framework Builder ## What this skill does This skill designs an evaluation framework for an LLM agent or pipeline. Most teams skip evals until something breaks in production — this skill helps you build evals before launch so you have a baseline, catch regressions, and measure quality improvements objectively. It covers dataset construction, metric selection, LLM-as-judge setup, and CI integration. ## How to use ### Claude Code / Cline Copy this file to `.agents/skills/agent-eval-framework-builder/SKILL.md` in your project root. Then ask: - *"Use the Agent Eval Framework Builder to design evals for our support chatbot."* - *"Build an evaluation suite for our RAG pipeline."* Provide: - What the agent does - What "good output" looks like - Sample inputs (5–10 examples if available) - Whether you have ground-truth answers or need to generate them ### Cursor / Codex Describe the agent and its task alongside these instructions. ## The Prompt / Instructions for the Agent When asked to build an evaluation framework, produce the following: ### Step 1 — Choose the right eval type | Agent Task | Eval Type | Reason | |---|---|---| | Factual Q&A with known answers | Exact match / F1 | Ground truth available | | Summarization, drafting | LLM-as-judge | No single right answer | | Code generation | Unit test execution | Correctness is verifiable | | Multi-step agent task | Trajectory scoring | Need to evaluate the path, not just the endpoint | | Classificatio...