← ClaudeAtlas

seed-corpuslisted

Turn this shell test suite into a replayable corpus, or seed a corpus from tests/<x>.test.sh with pending rows for the known gaps. Use when you need to capture and replay all the checks a suite exercises as data-driven test rows, so later changes can verify against exact payloads instead of reimplementing the original logic.
dimitritholen/1337-claude · ★ 0 · Data & Documents · score 62
Install: claude install-skill dimitritholen/1337-claude
You are seeding a corpus of captured test data: one JSON row per check, with the exact payload, exit code, and error message each check produces. This becomes a data-driven test suite that can run in isolation without reimplementing the original test's logic. Worked example: tests/fixtures/guard-corpus.jsonl and tests/guard-corpus.test.sh (sighting: task #657, commit ef0f163). # Procedure ## 1. Measure the suite at runtime Do not count checks by reading the test file. Checks in loops execute many times, and reading source code misses the total. Instrument or trace the `check()` helper (or equivalent) at runtime and count every execution. - Example: orchestrator-guard.test.sh has 125 lines of checks, but they execute 158 times because some sit in loops. ## 2. Capture one row per check execution For each executed check, capture the exact payload (as JSON), the exit code it produced, and a stderr fragment (the refusal message or error). Write one JSON object per line into tests/fixtures/<name>-corpus.jsonl, in the suite's original order. Row schema: `{gap, tool, input, want, stderr, env, pending, note}` - `gap`: empty string for rows captured from the suite; a ticket id for rows added later as known gaps. - `tool`: the tool name (e.g., `Write`, `Edit`, `Bash`, `Agent`). - `input`: the exact tool input JSON object (top-level keys: `tool_name`, `tool_input`, and optionally `agent_id` for subagent calls). - `want`: the exit code the check expects (0 for pass, non-zero for