← ClaudeAtlas

hybrid-search-eval-authorlisted

Use when a search system combines keyword (BM25) and vector retrieval, with or without a reranker, and needs numbers: judgment sets, nDCG@k and MRR on fused results, Reciprocal Rank Fusion vs weighted fusion vs single-stage, and whether the reranker earns its latency.
testland/qa-skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill testland/qa-skills
# hybrid-search-eval-author Hybrid retrieval (BM25 + dense vector + optional reranker) is now the dominant production pattern for semantic search and RAG pipelines. The sibling skill `vector-search-recall-tests` covers recall@k for the vector stage in isolation. This skill covers the fused result: does RRF outperform a weighted sum? Does the reranker improve nDCG@10 enough to justify the added latency? Per the [OpenSearch hybrid search blog], nDCG@10 improved an average of +12.08% over BM25 across seven BEIR datasets when using hybrid queries with normalization. The skill gives you the tests to verify - or disprove - that claim on your own corpus. ## When to use - Production pipeline combines BM25 (term) and kNN (dense) retrieval. - Choosing between fusion strategies: Elasticsearch RRF vs OpenSearch weighted fusion vs a custom reranker pass. - Quantifying reranker ROI: does cross-encoder/Cohere/bge lift nDCG enough to accept the latency increase? - Establishing a numeric relevance baseline before a model or index change (regression guard). ## Step 1 - Build a judgment set (qrels) Relevance evaluation requires graded relevance labels. Three methods, cheapest to most accurate: - **Proxy labels from click logs** (fastest): grade by engagement, e.g. clicked + dwell > 30s -> 2, clicked -> 1, impression only -> 0. - **LLM-assisted labeling** (cost-effective at scale): prompt an LLM judge to return a 0-3 grade per query-document pair. - **Human annotation via poolin