seo-ai-crawlers

# seo-ai-crawlers (M14) Two separate questions live in this module, and only one of them has vendor documentation behind it: 1. **Google's AI features.** A page can appear in AI Overviews / AI Mode only if it is indexed and eligible to be shown in Google Search **with a snippet**. Those gates are documented, so `M14.ai_eligibility.*` carries the only `established` findings here. 2. **Third-party AI crawlers.** Who may fetch and cite the page — OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta, Mistral, DuckDuckGo, Cohere. Access is documented per user-agent; the citation *effect* is not, so those findings stay `directional`. The training-vs-retrieval-vs-user-fetch distinction decides every robots recommendation. Reference: `references/ai-crawlers.md` (UA table with sources, Content-Signal, eligibility gates, presets). AI discovery files (llms.txt, agents.md, `/.well-known/ucp`, agentic sitemap) are module M21 — see `seo-ai-discovery`; agentic-commerce readiness is M18 (`seo-ecommerce`). ## Inputs Work from the PageSnapshot named in your dispatch envelope: read `parsed` from `<run_dir>/pages/<slug>.json` plus `robots_directives`, `headers` (`x-robots-tag`), and `render{needed,used,delta}`; Grep `pages/<slug>.html` for verbatim evidence. `<run_dir>/site/robots.json` carries, for the **audited path**, `verdicts`, `effective_access{[bot]:{class, fetch, url_allowed, via, rule, signals, doc_url, robots_reliability}}`, `content_signals{present, placement, global, by_group[], ma...