seo-technical

# Technical SEO Audit ## When to Use - Use when the user wants a technical SEO review focused on crawlability, indexability, performance, or rendering. - Use when auditing robots.txt, canonicalization, JavaScript SEO, Core Web Vitals, or AI crawler access. - Use when the task is infrastructure- and implementation-oriented rather than content-focused. ## Categories ### 1. Crawlability - robots.txt: exists, valid, not blocking important resources - XML sitemap: exists, referenced in robots.txt, valid format - Noindex tags: intentional vs accidental - Crawl depth: important pages within 3 clicks of homepage - JavaScript rendering: check if critical content requires JS execution - Crawl budget: for large sites (>10k pages), efficiency matters #### AI Crawler Management As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration. **Known AI crawlers:** | Crawler | Company | robots.txt token | Purpose | |---------|---------|-----------------|---------| | GPTBot | OpenAI | `GPTBot` | Model training | | ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing | | ClaudeBot | Anthropic | `ClaudeBot` | Model training | | PerplexityBot | Perplexity | `PerplexityBot` | Search index + training | | Bytespider | ByteDance | `Bytespider` | Model training | | Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) | | CCBot | Common Crawl | `CCBot` | Op...