← ClaudeAtlas

failure-to-eval-capturelisted

Iterate-stage skill: converts a production failure into a scrubbed, permanent eval case — PII removed with the failure's trigger structure demonstrably preserved, wired into regression testing. Use when a bad output escaped to production — 'capture this failure as an eval case', 'turn this incident into a permanent regression test', 'encode this so it never ships again' — or when /pm routes such a request here. Do NOT use for batch curation of reviewed outputs (golden-dataset-builder), for root-causing why the model failed, for PII scrubbing with no eval encoding, or for regression-testing definitions.
Abhillashjadhav/PM-agent-OS · ★ 1 · AI & Automation · score 60
Install: claude install-skill Abhillashjadhav/PM-agent-OS
# Failure-to-Eval Capture Every production failure becomes a permanent test — scrubbed of the people it happened to, but still carrying the trap it fell into. Scrub the identity, keep the mechanism. ## Verification gates (defined first; output is blocked until all pass) - **G1 — Scrubbed:** all PII and identifying detail (names, emails, orgs, distinctive figures) replaced with stable placeholders; a scrub table shows placeholder classes, never relisting the original values; the unscrubbed failure never appears in the eval artifact. - **G2 — Pattern demonstrably preserved:** the case states the failure mechanism and shows that the scrubbed input retains its trigger structure — an explicit preservation argument ("mechanism: entity-invention from org-name context; scrubbed input keeps org-as-topic + attendee-owned action item"). Over-scrubbing that deletes the trigger fails this half as hard as leaking PII fails the other. - **G3 — Mechanically assertable:** the expected-behavior assertion is checkable without a judge wherever possible (every attendee in output ∈ input attendee list); the case carries id, failure class, provenance (date/ticket, never customer identity), and its regression wiring. ## Steps 1. **Name the mechanism first.** What made the output wrong — not "bad summary" but "invented an entity from an org name mentioned as topic, then reassigned an action item to it". The scrub is designed around protecting this mechanism; naming it first is what makes G2 chec