ai-crawler-policylisted

# AI Crawler Policy The decision people think they're making is "should AI companies use my content". The decision they're actually making is four separate ones, and conflating them is how sites end up blocking the crawler that would have cited them while allowing the one that trains on them. ## The four jobs, and why the distinction matters | Job | What it does | Blocking it means | |---|---|---| | **Training** | Crawls to build model training corpora | Your content isn't in the next model. No effect on citations today | | **Search indexing** | Builds the retrieval index the assistant searches | The assistant can never find you. This is the expensive one | | **Live fetch** | Fetches a URL now because a user asked | The assistant can't read a page a user pasted | | **Classic search** | Googlebot, Bingbot | You leave search entirely. Never block these | Most people want to block training and allow retrieval. That's a coherent position and it's implementable. What isn't coherent is blocking retrieval and then asking why ChatGPT never mentions the brand. ## The agent table ``` # Training crawls GPTBot OpenAI training ClaudeBot Anthropic training Google-Extended Google Gemini training ONLY — does not affect AI Overviews Applebot-Extended Apple Apple Intelligence training CCBot Common Crawl public dataset, used by many trainers Meta-ExternalAgent Meta training