conformance-debuglisted

Fast, targeted debugging of the Harness Conformance Lab (conformance/) — use whenever a conformance check fails (locally or in CI), a check is recorded ADAPTED, a harness version drifted, a harness TUI check needs poking at directly, or you're about to guess at a fix instead of reproducing the real failure. Covers reading the failure's shape, checking the vendor's current docs/changelog/issues before touching code, the --sandbox and --experiment fast loops, proving the provider speaks the real wire shape with --discovery and mitmproxy, telling a vendor limitation from a Lab defect, and reading evidence/verdict.json.
hiukky/uze · ★ 1 · Code & Development · score 74

Install: claude install-skill hiukky/uze

# Debugging the Harness Conformance Lab The Lab runs the real vendor CLI (Claude Code, Codex, OpenCode, Antigravity) against a synthetic provider in a disposable Docker container. What every harness must prove lives in `conformance/contract/` (outcome terms, no vendor named); how each harness is driven lives in `conformance/harnesses/<vendor>/bindings.py`; what is genuinely unique to a vendor stays in its `scenarios.py`. A full gate run (`python3 conformance/lab.py --harness <h>`) takes 6-11 minutes and only tells you pass/fail per check. **Never iterate against that loop.** There are seconds-long loops built for exactly this — reach for them first. The Lab's whole value is that a green is real. Two things make a green false, and both have shipped for months before: a check that passes because nothing happened (a hook that never ran, a tool call the harness rejected before executing), and a provider that speaks a wire shape the real API does not, so the harness silently does something else than it would in production. Every step below exists to catch one of those. ## Step 0 — is the world still the one the Lab assumes? Do this **before** touching code, on any failure, any ADAPTED, and any `VERSION DRIFT` line in the run summary. The Lab pins nothing: the image installs each vendor's channel-latest, so a run that fails "for no reason" is usually the vendor moving. - **Read the vendor's current docs and changelog — the ones that shipped with the binary you are testing, n