← ClaudeAtlas

judgment-list-authorlisted

Use when a team must create or refresh the human relevance judgments that search tests rely on, before running NDCG, MRR or recall evaluations: query sets, grading scales, rater guidelines, inter-rater agreement, Quepid, TREC-style pooling and refresh cadence.
testland/qa-skills · ★ 0 · Testing & QA · score 72
Install: claude install-skill testland/qa-skills
# judgment-list-author Related skills (`elasticsearch-relevance-tests` - covering Elasticsearch, OpenSearch, and Solr - and `vector-search-recall-tests`) all require a judgment list - a set of `(query, document_id, grade)` triples that define what "relevant" means for your product. No automated metric creates that corpus. This skill does. Per [TREC's pooling methodology](https://trec.nist.gov/overview.html), "NIST pools the individual results, judges the retrieved documents for correctness, and evaluates the results" - the judgment list is the non-automated step that all automated metrics depend on. ## When to use - Before running `_rank_eval` or `recall@k` for the first time: no judgment list means no metrics. - After a major schema or corpus change: existing judgments cover documents that may no longer exist or new documents that are entirely unjudged. - When a relevance-regression check reports > 30% unrated docs across queries: the judgment pool is stale. - When adding a new query segment (new product category, new language) with no coverage in the existing set. ## Step 1 - Select the query set (head / torso / tail sampling) Not all queries deserve equal judgment effort. Sample across three tiers: | Tier | Definition | Suggested count | Priority | |---|---|---|---| | Head | Top ~100-500 by traffic volume (covers ~80% of impressions) | 50-100 queries | Highest | | Torso | Queries ranked ~500-5000 (navigational, faceted) | 100-200 queries | Medium | | Tail |