retrieval-evallisted

Measures the retrieval stage in isolation — loads or builds a labelled question→gold-chunk set, runs the project's OWN retriever, and reports recall@k, context precision/relevance, and filtered recall (recall with the tenant/permission predicate applied, where a pre-filter strands an HNSW traversal and recall craters silently), plus the retrieval-vs-generation split that says which stage actually failed. Also the tuning loop for top_k, chunk size, reranker on/off, and ef_search/nprobe. TRIGGER — before tuning any retrieval parameter, when a RAG answer is wrong and nobody knows which stage broke, and dispatched by /ai-audit, @rag-architect, @ai-feature-reviewer dimension 3, and /add-ai-feature Phase 7. ANTI-TRIGGERS (do NOT fire) — end-answer quality, faithfulness, or answer relevance (that is eval-run against the project's harness); ANN index configuration, params, metric, or refresh (that is vector-index-audit); a corpus with no labelled set (HALT with the construction recipe — never score a set the model wr
adnanmokhtar/refract · ★ 1 · AI & Automation · score 80

Install: claude install-skill adnanmokhtar/refract

# Skill: retrieval-eval ## Premise In a RAG system retrieval quality is the #1 failure mode, and an answer-only score cannot see it: a perfect prompt over the wrong chunks produces a confident wrong answer that scores badly for reasons nobody can attribute. This skill isolates the stage — it runs the project's **own** retriever over a labelled question→gold-chunk set and reports what came back, before any generation happens. **Every number cites its `k`, its filter state, and its dataset version.** `recall@10 = 0.82 (tenant filter applied, set v3, 40 questions)` is a measurement; "recall is good" is a vibe, and `recall = 0.82` with no `k` is not emittable. **A `PASS` requires a declared target** — the recall/latency/scale target is a `vector-store-ops` §3 obligation the project owes, not a number this skill invents. Where no target was declared, the verdict for that metric is `UNSTATED`, and the finding is the absence. This skill measures. It does not choose the chunker, the embedding model, or the index parameters — `@rag-architect` designs those, `vector-index-audit` audits them, and this skill is the loop they both tune against. ## Adapt to the codebase Run the project's retriever exactly as production calls it — same embedding model, same query prefix, same filters, same top-k, same reranker. A measurement taken through a hand-rolled second retriever measures the second retriever. | What to mirror | Where it lives | Why it must not be substituted | |---|---|---| |