agent-evaluation

# Agent Evaluation Evaluate observable agent behavior against task-specific cases. Modified by AAS maintainers on 2026-09-05 to remove unsupported benchmark claims, correct uncertainty/error reporting and separate optional architecture sketches from the operating procedure. ## When to Use Use when comparing a changed agent, prompt or tool configuration, reproducing an observed failure, or estimating reliability on a declared task distribution. Do not infer product readiness from a public benchmark percentage or a generic score threshold. ## Prerequisites - A versioned case set with expected observable outcomes and permission boundaries. - A known baseline and candidate revision, including model, prompt, tools, configuration and runtime versions. - Authorized synthetic or redacted inputs, isolated targets and a bounded token, time and cost budget. - A verifier that distinguishes wrong outcomes, expected safe rejections, evaluator failures and infrastructure outages. Provider access is needed only if the declared evaluation calls that provider. ## Evaluation procedure 1. **Freeze the contract.** Record case IDs and dataset revision, baseline/candidate identities, target environment, repeat plan, budgets, stopping rule and decision criteria before execution. Keep critical safety and authorization failures separate from average quality; they cannot be compensated by a higher score. 2. **Validate the harness.** Run a known-pass case, a known-fail case and a deliberate verif...