eval-design-forensics

# Eval-Design Forensics — does the evaluation measure what the paper claims? Audit evaluation-design and reporting validity for: **$ARGUMENTS** (requires `claims.json` from `/evidence-ledger`). Emit span-anchored `eval-design-forensics.findings.json`. This skill computes **no verdict**. > 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing input — it proposes the findings the deterministic adjudicator > turns into the report. Re-firing it on a wall-clock timer adds no signal: its > output changes only when the **paper / ledger** changes (or a repo arrives, > raising the observability level), not with the clock. Schedule the *external wait > that precedes it* — ledger built (or artifacts released → L2) → audit **once**. > (Mirrors ARIS's external-cadence doctrine.) > Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of > Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench > self-enhancement, self-preference, position bias), and the "Show Your Work" / > reproducibility-checklist reporting norms — reframed to audit a **third party's** > evaluation. A favourite autoresearch shortcut is to report a number that is > arithmetically self-consistent (family A), runs real code against a real ground > truth (family D), and **still does not measure what it claims**: the protocol > leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the > reporting quietly drops a declared condit...