seo-crawlability

# seo-crawlability (M1) Crawl access is the precondition for every other search signal: if Googlebot/Bingbot can't fetch the page and its assets, nothing else ranks. This module covers general-purpose crawl access only. AI-specific bot directives (GPTBot, Claude-SearchBot, etc.) live in `seo-ai-crawlers` (M14) and AI discovery files (`llms.txt`, `agents.md`, UCP) in `seo-ai-discovery` (M21); see `references/ai-crawlers.md` for that boundary. ## Inputs Work from the PageSnapshot named in your dispatch envelope: read `parsed` from `<run_dir>/pages/<slug>.json` plus `status_chain` and `headers`, and the parsed robots file in `<run_dir>/site/robots.json` (syntax, groups, Googlebot/Bingbot verdicts for the audited URL, `Sitemap:` lines); Grep `pages/<slug>.html` for verbatim evidence. Deterministic findings already emitted by `audit.mjs` are listed in `<run_dir>/findings.deterministic.json` — do not re-emit those ids; add model-judged findings only. If invoked directly with a URL/path and no snapshot exists, first run `node "${CLAUDE_PLUGIN_ROOT}/scripts/snapshot.mjs" <target> --out "${CLAUDE_PLUGIN_DATA}/runs"` and use the printed snapshot path. ## Audits Working from the PageSnapshot (`parsed_rendered` when `render.used` is not `none`, else `parsed`) plus `site/robots.json`: 1. **Reachability**: `/robots.txt` returns 200 (a 404 means "allow all" but is worth flagging; a 5xx can suspend crawling). 2. **Syntax**: each line is a valid directive (`User-agent`, `Disallow`, `Allow`...