agent-evaluation

# Agent Evaluation Build evidence that can inform a release owner, not a showcase of favorable examples or a safety certification. ## Use when - Define quality before building or changing an agent. - Compare prompts, models, tools, memory strategies, or orchestration patterns. - Convert production failures into regression cases. - Establish a repeatable release gate or human-review plan. ## Inputs Collect the agent objective, users, supported tasks, unacceptable outcomes, current baseline, execution environment, available traces, and evaluation budget. State assumptions when an input is unavailable. ## Output contract Produce: 1. An evaluation brief with scope, risks, hypotheses, and frozen system versions. 2. A dataset manifest with provenance, categories, splits, and contamination controls. 3. A scoring specification with rubrics, graders, thresholds, and tie-breaking rules. 4. Reproducible run settings, aggregate results, uncertainty, and baseline deltas. 5. A failure taxonomy, representative cases, evidence limits, and a decision memo for the accountable release owner. ## Workflow 1. Define the unit under test and the decision the evaluation must support. Separate model quality from tool, retrieval, policy, and infrastructure failures. 2. Convert user goals and risks into observable criteria. Include task success, safety, latency, cost, and escalation quality only when relevant. 3. Build representative cases from real distributions where permitted. Add boundary,...