ai-crawler-accesslisted

# AI crawler access ## Goal A site is cited only by engines that can read it, and most robots.txt files were written before AI crawlers existed. This skill turns the default (silence, which some bots read generously and others don't) into explicit per-bot decisions the user actually made. It edits the working directory and spends nothing. ## Required inputs - The site source containing `robots.txt` (or the place it should exist). If the file is generated, find the generator — editing build output fixes nothing. ## Workflow 1. Read the current `robots.txt`. Note every existing rule; nothing here removes a rule the user wrote on purpose. 2. Put the decision to the user per crawler — what each one feeds is the whole tradeoff: | User-agent | Feeds | Blocking it means | |---|---|---| | `GPTBot` | OpenAI training and search index | ChatGPT is less likely to know or cite the site | | `ClaudeBot` | Anthropic training and retrieval | Claude is less likely to know or cite the site | | `PerplexityBot` | Perplexity's answer index | Absent from Perplexity answers | | `Google-Extended` | Gemini training only | Google Search and its rankings are unaffected | | `CCBot` | Common Crawl, which many models train on | Slowly absent from models trained on it | A site that wants citations allows the answer-feeding bots; blocking `Google-Extended` or `CCBot` is a defensible training-only stance. Both are choices — record whichever the user makes. 3. Write each decisio