evaluate-an-ai-systemlisted

# evaluate-an-ai-system Build a release decision around representative work, not a few favorable demos. Keep the evaluation small enough to run often, rich enough to expose important failures, and versioned so improvements can be distinguished from a changing test. ## When to use Use this skill before shipping or changing an AI assistant, classifier, extraction system, retrieval feature, agent workflow, or model-backed decision aid. Use it when two versions feel different but the team cannot explain whether users are better served. Do not place private user data, copyrighted corpora without permission, or unsafe content into an evaluation set casually. Do not let an automatic judge make the final decision for high-impact work. ## Preconditions - Name the user task, release owner, risk owner, intended users, and decisions the system may influence. - Collect allowed examples from real task shapes and document how sensitive fields are removed or protected. - Record the current production or manual process as the baseline. - Define unacceptable failures and the person authorized to decide borderline cases. ## Procedure 1. Break the user task into capabilities and failure slices: common, rare, ambiguous, adversarial, multilingual, long-context, missing-data, tool-failure, and safety-sensitive cases where relevant. 2. Build a versioned evaluation set with input, permitted context, expected properties, risk tag, and source. Keep a held-out portion away from day-to-day