ai-crawlerslisted
Install: claude install-skill sitetiller/ai-visibility-skills
# AI crawler access
Goal: tell the user exactly which crawlers can read their site, which rule decides it, and what to change so their choice is deliberate: usually **search and answer crawlers allowed, training crawlers the user's call**.
## Inputs
- A site or page URL, or a local robots.txt draft.
- Optional: the user's policy ("stay in AI search, stay out of training", "block everything", "allow everything").
## Steps
1. Run the live check (add a path to test a section):
```bash
npx sitetiller crawlers example.com --json
npx sitetiller crawlers example.com/docs/ --json
```
2. Read each bot's `allowed`, `rule`, `line`, `group` and `purpose`. Group findings by purpose:
- **search**: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocked = cannot be cited in that engine's answers.
- **user**: ChatGPT-User, Claude-User, Perplexity-User. Fetches a person asked for. OpenAI and Perplexity say robots.txt may not apply.
- **training**: GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended (tokens, not crawlers). Blocking does not remove the site from those vendors' search features.
3. If a search crawler is blocked and the user did not choose that, write the fix:
- Blocked by its **own named group** → edit that group (the line number is in the output); do not add a second group for the same bot.
- Blocked by the **`*` group** → add a named group above it. A named group replaces the `*` rules for that bot, so copy any `*`