ai-crawler-accesslisted
Install: claude install-skill visibian/seo-geo-skills
# AI crawler access
Help the user decide, check and write which AI crawlers may read their site. Each AI company runs separate crawlers for separate jobs, and blocking the wrong one can take a site out of an assistant's answers.
## The three kinds of crawler
- Search crawlers fetch pages so an assistant can find and cite them in answers (OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, Googlebot for Google Search with AI Overviews and AI Mode, Bingbot for Bing and Microsoft Copilot).
- User fetchers open a page when a person asks an assistant about it (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, MistralAI-User). Some do not always follow robots.txt.
- Training crawlers and settings collect pages to train models (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training, CCBot).
Blocking training does not remove a site from search answers. OpenAI, for example, treats GPTBot and OAI-SearchBot as separate settings. The full list, with what each crawler does, what blocking it changes and each company's documentation, is in `references/ai-crawlers.md`. Read it before advising on a specific crawler.
## Steps
1. Get the file. In a codebase, look for `robots.txt` in the public or static folder, or for code that generates it (for example `app/robots.ts` in Next.js, a `robots` setting in a CMS, or a server rule). Otherwise fetch it from the live site.
2. Read it with the script, from this skill