judgment-list-authorlisted
Install: claude install-skill testland/qa-skills
# judgment-list-author
Related skills (`elasticsearch-relevance-tests` - covering Elasticsearch,
OpenSearch, and Solr - and `vector-search-recall-tests`) all require a
judgment list - a set of `(query, document_id, grade)` triples that define what
"relevant" means for your product. No automated metric creates that
corpus. This skill does.
Per [TREC's pooling methodology](https://trec.nist.gov/overview.html), "NIST
pools the individual results, judges the retrieved documents for correctness,
and evaluates the results" - the
judgment list is the non-automated step that all automated metrics depend on.
## When to use
- Before running `_rank_eval` or `recall@k` for the first time: no judgment
list means no metrics.
- After a major schema or corpus change: existing judgments cover documents
that may no longer exist or new documents that are entirely unjudged.
- When a relevance-regression check reports > 30% unrated docs across
queries: the judgment pool is stale.
- When adding a new query segment (new product category, new language) with
no coverage in the existing set.
## Step 1 - Select the query set (head / torso / tail sampling)
Not all queries deserve equal judgment effort. Sample across three tiers:
| Tier | Definition | Suggested count | Priority |
|---|---|---|---|
| Head | Top ~100-500 by traffic volume (covers ~80% of impressions) | 50-100 queries | Highest |
| Torso | Queries ranked ~500-5000 (navigational, faceted) | 100-200 queries | Medium |
| Tail |