llm-eval-anti-patternslisted
Install: claude install-skill testland/qa-skills
# llm-eval-anti-patterns
An LLM eval suite can be green and still be worthless. The failure is not in
the system under test, it is in the eval definition: too few cases to
distinguish signal from sampling noise, assertions that cannot fail for the
right reason, no baseline to compare against, or a judge model whose verdicts
have never been checked against a human.
This catalog reviews the **eval definition**, not the eval run. Read the
config, the test data, the assertions, and the CI job that executes them.
Every finding must point at a line in one of those.
## What this owns, and what it does not
| This catalog | Not this catalog |
|---|---|
| Reviews a suite that already exists | Building the suite, choosing a harness, or writing the first test cases |
| Judges whether the suite's design can produce a trustworthy verdict | Choosing which quality metrics matter for the product |
| Reports methodology defects with a severity and a fix | Ranking one model against another, or running the benchmark |
If there is no eval suite yet, there is nothing here to review. If the
question is "which model is better", that is a benchmarking exercise, and this
catalog only tells you whether the benchmark you are about to trust was
constructed soundly.
## Severity scheme
Three levels. These are a review convention, not a standard:
| Severity | Means |
|---|---|
| **Critical** | Silent regression risk, or eval theater: the suite passes for a reason unrelated to the behavior it claims