← ClaudeAtlas

qualitylisted

Prime on Eidet's quality and benchmarking surfaces before changing them — QualityService's check battery and QualityReport behind /api/eidet/quality, the deterministic retrieval scorecard (BenchmarkRunner's fused-vs-flat-baseline arms, RetrievalMetrics, AmaCapability headings, the CI-asserted docs/benchmark.md), and the SWE Context Bench harness in tools/Eidet.Bench with its LeaderboardGuard. Use when the task touches a quality check, a retrieval metric, the gold dataset, eidet bench, or a benchmark number. Not for the ranking pipeline itself (see recall), not for the integrity auditor (see writepath).
stevehansen/eidet · ★ 4 · Data & Documents · score 70
Install: claude install-skill stevehansen/eidet
# Quality & benchmarking — priming **Canonical spec:** `docs/domains/quality.md` — read it for the check battery, the harness shape, all invariants, key files, and gotchas. Generated artifacts: `docs/benchmark.md`, `docs/swe-context-bench.md`. Two questions: "is this repo's memory healthy?" (`QualityService`, advisory and read-only) and "does our ranking actually beat the baseline?" (deterministic scorecard + the SWE Context Bench harness). ## Core invariants (get these right) - **A leaderboard-shaped number may only come from the real dataset.** `LeaderboardGuard` refuses for fixtures, and the rendered fixture report carries a "not a leaderboard" banner. - **`ToMarkdown` is pure** (no timestamps/environment) so CI can assert `docs/benchmark.md` byte-equal. Regenerate via the documented env var; never hand-edit. - **Both arms run over the same candidate pools** — the baseline must differ by ranking alone. - **Report un-scoreable capabilities as not-evaluated, with the reason** — never fabricate. - **Metrics define edge cases out of existence**: empty gold → 0, `k` clamps, duplicates can't exceed 1.0, everything finite in `[0,1]`. - **The benchmark calls `RecallScoring.Fuse` directly**, which is exactly why production trust/ROI policy stays outside `Fuse`. - **The quality report never mutates anything** — it surfaces what other domains recorded. - **The analysed sample is bounded** (`TotalMemories` vs `AnalyzedCount`), and the integrity auditor is optional — a cl