agent-evaluation-engineeringlisted

# AI Agent Evaluation Engineering Use this skill when creating or reviewing an evaluation suite for an AI agent. Treat an evaluation as an executable product contract, not a collection of impressive demos. ## 1. Define the contract and risk Write down the user goal, allowed inputs, observable outputs, available tools, permissions, side effects, latency and cost limits, and acceptable escalation behavior. Rank failure modes by impact and likelihood. Keep capability quality separate from trust and safety: an agent can complete a task and still leak data, exceed authority, or make an irreversible change. ## 2. Build representative cases Create cases from real user journeys, production failures, domain edge cases, ambiguous requests, missing context, tool outages, permission boundaries, prompt injection, and unsafe requests. Include single-turn and multi-turn cases where memory, correction, or recovery matters. Partition cases into: - development examples for rapid iteration; - a held-out regression set that prompts and graders do not see during tuning; - adversarial cases focused on authority, privacy, injection, and irreversible effects. Turn every confirmed production failure into a minimized regression case. Remove duplicates and document why each case exists. ## 3. Specify evidence and graders For each case, define the expected invariant before running it. Prefer observable evidence: - final answer facts and citations; - tool names, arguments, results, ordering, r