← ClaudeAtlas

automation-comparelisted

Verifies that generated Playwright tests actually implement the scenarios they claim to, step by step, and reports a match percentage plus a regenerated requirements traceability matrix. Use this skill after automation code is generated or changed, whenever the user asks what the automated suite really covers, whenever test cases in the TMS and the code have drifted apart, or before uploading cases to a TMS. Also use it to audit an inherited suite whose coverage nobody can vouch for.
YevheniiaQA/qa-agent-toolkit · ★ 0 · AI & Automation · score 72
Install: claude install-skill YevheniiaQA/qa-agent-toolkit
# Automation Compare The classic decay: test cases live in the TMS, tests live in the repo, and after three sprints nobody knows which describes reality. This skill is the guard against that drift — scenario JSON is the source of truth, and code is checked back against it. ## What to compare For every test case in the scenario JSON, locate its implementation (by scenario ID in the test title) and check, step by step: | Scenario element | Check against code | |---|---| | Preconditions | Present as fixture, setup call, or `beforeEach` | | Each step's action | The corresponding request or interaction exists, same method/endpoint/element | | Each step's input data | Same values, or a documented equivalent (env var, fixture) | | Each step's expected result | A real assertion exists for it — not an adjacent one | | Postconditions | Verified, or explicitly waived with a reason | | Priority | Reflected in the tag (`@smoke` for HIGH) | An expected result with no corresponding assertion is the most common and most consequential gap: the step executes, nothing is checked, the test passes forever. ## Scoring ``` step coverage = asserted steps / total steps case coverage = implemented cases / total cases suite match = weighted mean, HIGH priority cases weighted 2× ``` Report per suite and overall. Below 90% suite match, hand the gaps back to the automation skill for a targeted regeneration pass rather than a full rewrite — regenerating everything discards the debugging wor