llm-crawler-access-checklisted

# AI crawler access check One wrong line in `robots.txt` removes a site from AI answers completely, and no amount of content work can compensate. This check takes under a minute and should run before any other AI-visibility work. ## Scope Read `https://<domain>/robots.txt` and nothing else. Do not crawl the site, do not attempt to access disallowed paths, and do not bypass any access control. This is a read of one public file. ## Procedure ### 1. Fetch Fetch `https://<domain>/robots.txt`. - **404 or empty** - everything is allowed by default. Say so; that is a valid and often correct configuration. Stop and report. - **Non-200 other than 404, or unreachable** - report the status code and stop. Do not guess at contents. - **Served as HTML** (a soft 404 returning the site's error page) - flag it. Crawlers may parse it as garbage. This is itself a finding. ### 2. Resolve each agent For each agent below, apply standard robots.txt matching: the most specific `User-agent` group that names the agent wins, and `*` applies only when no group names it. Within the winning group, the longest matching path rule wins, and `Allow` beats `Disallow` on an equal-length match. | Agent | Operator | Purpose | What blocking it actually costs | | --- | --- | --- | --- | | `OAI-SearchBot` | OpenAI | search index | citations in ChatGPT Search | | `ChatGPT-User` | OpenAI | live fetch during a chat | the model cannot open your page when a user asks about it | | `GPTBot` | OpenAI | train