← ClaudeAtlas

rule-gate-integritylisted

Four ways a gate, test, or generator check passes while proving nothing: grading a copy of itself, passing on emptiness, a canary firing for the wrong reason, and a summary line read as a verdict. Load before writing a gate, a mutation harness, or any check that guards generated output.
djnsty23/claude-auto-dev · ★ 3 · AI & Automation · score 72
Install: claude install-skill djnsty23/claude-auto-dev
# A gate that cannot fail is not a gate These four failure modes were hit independently by two sessions on the same day, working on unrelated problems — a mutation harness for a token generator, and a test-vacuity sweep across a plugin marketplace. Both arrived here the hard way. Each one produces a **green result that means nothing**, and each is invisible from the summary line. ## 1. Run the real thing. Never grade a copy. A check that rebuilds what it is checking grades its own reconstruction. It passes happily while the shipped artefact emits something else. - A token gate must **run the real generator in `--check`** and read its exit code, not assemble the CSS it expects and diff that. - A retry-policy test must **import the real function**. One in a production repo reimplemented `withTransientRetry` inline; the copy was faithful when compared, which is the most dangerous state for a copy to be in — it looks like evidence. If speed is the reason for the copy, make the *slow part* injectable instead. A 2s/8s/32s backoff became an optional parameter that production never passes; the real function then runs in milliseconds under test. The same trap has a timing form: a gate that regenerates an artefact and *then* reads it compares the generator against its own output and is green forever. One production repo shipped a stale manifest twice that way, with preflight passing both times. Snapshot what was **on disk** — what a commit would actually have shipped — befo