← ClaudeAtlas

ai-crawler-accesslisted

Check and write the robots.txt rules that decide which AI crawlers can read a website (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot, Meta, Amazon, Mistral and more). Use when someone asks whether ChatGPT, Claude, Gemini, Perplexity or Copilot can read their site, whether to block GPTBot or AI training, why an assistant cannot see their pages, or wants robots.txt lines for AI crawlers.
visibian/seo-geo-skills · ★ 0 · AI & Automation · score 75
Install: claude install-skill visibian/seo-geo-skills
# AI crawler access Help the user decide, check and write which AI crawlers may read their site. Each AI company runs separate crawlers for separate jobs, and blocking the wrong one can take a site out of an assistant's answers. ## The three kinds of crawler - Search crawlers fetch pages so an assistant can find and cite them in answers (OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, Googlebot for Google Search with AI Overviews and AI Mode, Bingbot for Bing and Microsoft Copilot). - User fetchers open a page when a person asks an assistant about it (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, MistralAI-User). Some do not always follow robots.txt. - Training crawlers and settings collect pages to train models (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training, CCBot). Blocking training does not remove a site from search answers. OpenAI, for example, treats GPTBot and OAI-SearchBot as separate settings. The full list, with what each crawler does, what blocking it changes and each company's documentation, is in `references/ai-crawlers.md`. Read it before advising on a specific crawler. ## Steps 1. Get the file. In a codebase, look for `robots.txt` in the public or static folder, or for code that generates it (for example `app/robots.ts` in Next.js, a `robots` setting in a CMS, or a server rule). Otherwise fetch it from the live site. 2. Read it with the script, from this skill