← ClaudeAtlas

ai-crawlerslisted

Check and fix which search and AI crawlers can reach a site: OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot and more, grouped by purpose (search, user fetch, training). Use when the user asks 'can ChatGPT/Claude/Perplexity see my site', 'should I block GPTBot', 'why isn't my site cited by AI', 'write my robots.txt for AI', 'is Cloudflare blocking AI bots', or wants to test a robots.txt draft. Uses the sitetiller CLI against the live robots.txt or a local draft.
sitetiller/ai-visibility-skills · ★ 0 · AI & Automation · score 75
Install: claude install-skill sitetiller/ai-visibility-skills
# AI crawler access Goal: tell the user exactly which crawlers can read their site, which rule decides it, and what to change so their choice is deliberate: usually **search and answer crawlers allowed, training crawlers the user's call**. ## Inputs - A site or page URL, or a local robots.txt draft. - Optional: the user's policy ("stay in AI search, stay out of training", "block everything", "allow everything"). ## Steps 1. Run the live check (add a path to test a section): ```bash npx sitetiller crawlers example.com --json npx sitetiller crawlers example.com/docs/ --json ``` 2. Read each bot's `allowed`, `rule`, `line`, `group` and `purpose`. Group findings by purpose: - **search**: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocked = cannot be cited in that engine's answers. - **user**: ChatGPT-User, Claude-User, Perplexity-User. Fetches a person asked for. OpenAI and Perplexity say robots.txt may not apply. - **training**: GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended (tokens, not crawlers). Blocking does not remove the site from those vendors' search features. 3. If a search crawler is blocked and the user did not choose that, write the fix: - Blocked by its **own named group** → edit that group (the line number is in the output); do not add a second group for the same bot. - Blocked by the **`*` group** → add a named group above it. A named group replaces the `*` rules for that bot, so copy any `*`