ai-evalslisted

Use when testing an LLM-backed feature, prompt, tool loop or multi-step agent, where the same input can produce different outputs and a prompt or model change can regress behaviour with no code diff. The behaviour spec that precedes the prompt, scenario datasets including adversarial and degradation classes, deterministic assertions over OpenTelemetry traces, calibrated LLM-as-judge, CI gates with baselines, human-in-the-loop, and the production scoring loop that turns incidents into scenarios.
konradcinkusz/architecture-standards · ★ 0 · AI & Automation · score 72

Install: claude install-skill konradcinkusz/architecture-standards

# Evaluating LLM-backed features and agents **Read [`references/AI-EVALS.md`](references/AI-EVALS.md) before applying any of this.** That file is the standard; everything below it is a summary to help you decide whether this skill applies and to check your work afterwards. Reference-architecture principles: P13. ## What this standard covers - The model in one paragraph - The spec comes first - Scenarios are data - Layer 1: deterministic assertions on traces - Layer 2: LLM-as-judge - Gates - Production scoring closes the loop - Human-in-the-loop - Prove the suite can fail ## Failure modes | Symptom | Cause | |---|---| | Suite green, agent visibly worse in production | Scenarios cover the demo paths; no incident-to-scenario loop; judge grading fluency, not grounding | | Constraint scenario "passes" on a broken agent | Guard-then-bail before the assertion; the agent did nothing and the eval skipped | | Eval results differ run to run with no change | Nondeterminism unpinned: temperature, model minor version, or fixture state not reset between scenarios | | Judge scores jump after a quiet week | Judge model or prompt changed without a suite version bump and re-baseline | | Judge and users disagree about quality | Judge never calibrated against human labels; rubric anchors too vague to grade against | | Prompt edit ships with no eval run | Change detection watches code paths only; prompts and agent definitions not mapped as eval-triggering paths | | Adversarial scenarios pass