qa-agent-testing

# QA Agent Testing (Jan 2026) Design and run reliable evaluation suites for LLM agents/personas, including tool-using and multi-agent systems. ## Default QA Workflow 1. Define the Persona Under Test (PUT): scope, out-of-scope, and safety boundaries. 2. Define 10 representative tasks (Must Ace). 3. Define 5 refusal edge cases (Must Decline + redirect). 4. Define an output contract (format, tone, structure, citations). 5. Run the suite with determinism controls and tool tracing. 6. Score with the 6-dimension rubric; track variance across reruns. 7. Log baselines and regressions; gate merges/deploys on thresholds. Use the copy-paste templates in `assets/` for day-0 setup. ## Determinism and Flake Control - Control inputs: pin prompts/config, fixtures, stable tool responses, frozen time/timezone where possible. - Control sampling: fixed seeds/temperatures where supported; log model/config versions. - Record tool traces: tool name, args, outputs, latency, errors, retries, and side effects. ## Two-Layer Evaluation (2026) Evaluate reasoning and action layers separately: | Layer | What to Test | Key Metrics | |---------------|-------------------------------------|-----------------------------------------------------| | **Reasoning** | Planning, decision-making, intent | Intent resolution, task adhesion, context retention | | **Action** | Tool calls, execution, side effects | Tool call accuracy, compl...