geo-crawlers

# AI Crawler Access Analysis Skill ## Purpose This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO. ## Key Insight As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results. --- ## Complete AI Crawler Reference ### Tier 1: Critical for AI Search Visibility (RECOMMEND: ALLOW) These crawlers power the AI search products where users actively look for answers. Blocking them directly reduces your visibility in AI-generated responses. #### GPTBot - **Operator:** OpenAI - **User-Agent:** `GPTBot` - **Full User-Agent String:** `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)` - **Purpose:** Fetches content for ChatGPT's web browsing, plugins, and search features. Content accessed by GPTBot may be used to improve OpenAI models. - **Impact of Blocking:** Content will NOT appear in ChatGPT Search results or be accessible when users a...