hybrid-search-eval-authorlisted
Install: claude install-skill testland/qa-skills
# hybrid-search-eval-author
Hybrid retrieval (BM25 + dense vector + optional reranker) is now the
dominant production pattern for semantic search and RAG pipelines. The
sibling skill `vector-search-recall-tests` covers recall@k for the
vector stage in isolation. This skill covers the fused result: does RRF
outperform a weighted sum? Does the reranker improve nDCG@10 enough to
justify the added latency?
Per the [OpenSearch hybrid search blog], nDCG@10 improved an average of
+12.08% over BM25 across seven BEIR datasets when using hybrid queries
with normalization. The skill gives you the tests to verify - or
disprove - that claim on your own corpus.
## When to use
- Production pipeline combines BM25 (term) and kNN (dense) retrieval.
- Choosing between fusion strategies: Elasticsearch RRF vs OpenSearch
weighted fusion vs a custom reranker pass.
- Quantifying reranker ROI: does cross-encoder/Cohere/bge lift nDCG
enough to accept the latency increase?
- Establishing a numeric relevance baseline before a model or index
change (regression guard).
## Step 1 - Build a judgment set (qrels)
Relevance evaluation requires graded relevance labels. Three methods,
cheapest to most accurate:
- **Proxy labels from click logs** (fastest): grade by engagement, e.g.
clicked + dwell > 30s -> 2, clicked -> 1, impression only -> 0.
- **LLM-assisted labeling** (cost-effective at scale): prompt an LLM
judge to return a 0-3 grade per query-document pair.
- **Human annotation via poolin