orchestrate-evidence-retrieval-expertlisted

# Orchestrate: Evidence Retrieval Expert **Evidence tier: first-hand build (August 2026).** Grounded in a completed Orchestrate submission that was audited to destruction — 48 logged defects, 9 measured-and-rejected optimisations, 17 certification scripts. Every number below was measured on that system. Nothing here claims access to HackerRank's internal scoring. ## The rule **Compute the ceiling before you optimise. Then benchmark the fashionable option and publish the number when it loses.** ## Step 1 — the pool bounds everything No ranker can retrieve an id that is not in the pool it ranks. Measure this first: ``` rule-scoped pool 26/31 = 83.9% <- what ships all-user pool 31/31 = 100% <- higher ceiling ``` Tempting. But measured end to end, the wider pool scored **F1 0.483 vs 0.512** — the extra candidates cost more precision than the recovered recall was worth. **A higher ceiling is not a higher score.** ## Step 2 — benchmark the fashionable option 153 configurations (3 pools × 17 rankers × 3 values of k), scored only on labeled rows: | ranker | F1 | |---|---| | BM25 | **0.512** | | tf-idf / jaccard / hybrid | 0.496 | | dense embeddings, RRF, cross-encoder | **0.479** | | recency only | 0.336 | **Every neural method lost to plain lexical matching.** The relation being scored was topical word overlap, not paraphrase — bi-encoders are built for the wrong thing here. Also measured: temporal, metadata and behaviour re-ranking produced **byte-identical**