golden-dataset-builderlisted

Iterate-stage skill: curates real outputs plus human judgments into golden eval cases — every case carrying the human verdict and the human's reason, no unlabeled cases, ever. Use when reviewed outputs need structuring into a reusable test set — 'build a golden dataset from these outputs', 'turn these human-graded examples into golden cases', 'we need a regression set, here are outputs and verdicts' — or when /pm routes such a request here. Do NOT use to grade the outputs (human judgment is input), to capture a single production failure (failure-to-eval-capture), to generate synthetic cases, or for golden-dataset definitions.
Abhillashjadhav/PM-agent-OS · ★ 1 · Data & Documents · score 60

Install: claude install-skill Abhillashjadhav/PM-agent-OS

# Golden Dataset Builder Real outputs, real human judgments, structured to last. A golden case without its human verdict and reason isn't golden — it's sand. ## Verification gates (defined first; output is blocked until all pass) - **G1 — No unlabeled cases:** every golden carries the human verdict AND the human's reason. Cases missing either are quarantined with the specific ask ("needs verdict", "'fine I guess' — pass or fail, and why?") — never included, never auto-labeled, never upgraded from ambiguity. - **G2 — Labels are the human's, verbatim:** reasons are stored as written; paraphrasing a reason into something stronger or cleaner corrupts provenance. Ambiguous verdicts go back as questions, not interpretations. - **G3 — Set integrity:** duplicates collapsed to one case with provenance noted; the set report shows candidates → goldens + quarantined, verdict balance (a set of only passes tests nothing), and failure-pattern coverage. ## Steps 1. **Inventory the raw material:** every output-review pair, however informal. Nothing is discarded — cases split into golden-ready (verdict + reason) and quarantine (missing pieces, each with its unblock question). 2. **Structure each golden:** id · input (or reference) · output · verdict · reason (verbatim) · label author · date · the eval criterion it exercises (mapped where an eval exists; `unmapped` honestly otherwise — unmapped goldens are seeds for eval-engine, noted as such). 3. **Deduplicate** on output identity; keep o