← ClaudeAtlas

llm-eval-anti-patternslisted

Use for any question about mistakes in LLM evals or auditing an eval suite that gates a release, model swap or prompt change, including evals that pass while users say the model got worse. Checks eight methodology errors (too few cases, exact match on open text, no baseline, unpinned models, no adversarial cases and more) and how to validate an LLM judge.
testland/qa-skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill testland/qa-skills
# llm-eval-anti-patterns An LLM eval suite can be green and still be worthless. The failure is not in the system under test, it is in the eval definition: too few cases to distinguish signal from sampling noise, assertions that cannot fail for the right reason, no baseline to compare against, or a judge model whose verdicts have never been checked against a human. This catalog reviews the **eval definition**, not the eval run. Read the config, the test data, the assertions, and the CI job that executes them. Every finding must point at a line in one of those. ## What this owns, and what it does not | This catalog | Not this catalog | |---|---| | Reviews a suite that already exists | Building the suite, choosing a harness, or writing the first test cases | | Judges whether the suite's design can produce a trustworthy verdict | Choosing which quality metrics matter for the product | | Reports methodology defects with a severity and a fix | Ranking one model against another, or running the benchmark | If there is no eval suite yet, there is nothing here to review. If the question is "which model is better", that is a benchmarking exercise, and this catalog only tells you whether the benchmark you are about to trust was constructed soundly. ## Severity scheme Three levels. These are a review convention, not a standard: | Severity | Means | |---|---| | **Critical** | Silent regression risk, or eval theater: the suite passes for a reason unrelated to the behavior it claims