ai-bot-log-analysislisted

Analyse server logs to see which AI and search crawlers actually visit a site, what they fetch, what they get, and where crawl budget goes. Use whenever the user mentions server logs, log file analysis, access logs, crawl budget, bot traffic, GPTBot or ClaudeBot or PerplexityBot hits, verifying crawler identity, fake bot traffic, or wants to know whether AI crawlers are reaching their content. Also use when pages aren't being indexed and the cause is unclear, or when bot traffic is affecting server load or costs.
keupera/seo-skills · ★ 0 · AI & Automation · score 72

Install: claude install-skill keupera/seo-skills

# AI and Search Bot Log Analysis Logs are the only source that tells you what actually happened rather than what should have happened. Search Console reports Google's view after processing. Logs show every request, from every agent, with the status you actually returned. For AI crawlers specifically, logs are the only source at all. None of the AI companies provide a Search Console equivalent, so if you want to know whether ChatGPT's indexer has ever fetched your pricing page, the log is the answer. ## What you need A standard combined-format access log, from the origin or the CDN. CDN logs are better because they see requests the origin never does, including the ones the CDN blocked, which is frequently the finding. ``` 93.184.216.34 - - [07/Sep/2026:14:22:11 +0000] "GET /pricing HTTP/1.1" 200 18422 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" ``` ## Which agents to look for ``` # Search Googlebot Google web search, AI Overviews, AI Mode Bingbot Bing and Copilot # AI search indexing — these are the ones that produce citations OAI-SearchBot OpenAI, ChatGPT search index PerplexityBot Perplexity index Claude-SearchBot Anthropic search index # AI live fetch — a user asked about this URL right now ChatGPT-User OpenAI Perplexity-User Perplexity Claude-User Anthropic # AI training GPTBot OpenAI ClaudeBot Anthrop