seo-log-analysislisted

# Server Log Analysis Analyzes local server log files for crawl budget breakdown. No MCP or external calls required. ## Inputs - `file`: Absolute path to server log file (Apache Combined, Apache Common, or Nginx access log). If user provides relative path, resolve with `Bash: realpath <path>`. ## Execution **Step 1: Format Detection** Read the first 10 lines of the log file to detect format: - Apache Combined: `%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"` — 9+ fields, has referer and UA in quotes - Apache Common: `%h %l %u %t "%r" %>s %b` — 7 fields, no referer/UA - Nginx: similar to Apache Combined with slight field order differences - Check for compressed files (.gz) — if detected, inform user to decompress first **Step 2: Parse Log Lines** Use Bash awk to extract fields. For Apache Combined/Nginx format (9 fields): ```bash awk '{ ip=$1; method_url=$7; status=$9; ua=$0 match($0, /"([^"]+)"$/, arr) # Extract UA from last quoted field print ip, $7, $9, arr[1] }' logfile ``` For Apache Common (7 fields): ip=$1, request=$7, status=$9, ua="unknown" **Step 3: Classify User-Agents** Group each request into categories: - **Googlebot**: `Googlebot`, `Googlebot-Image`, `Googlebot-News`, `AdsBot-Google` - **Bingbot**: `bingbot`, `BingPreview`, `MicrosoftPreview` - **Other search bots**: `Slurp` (Yahoo), `DuckDuckBot`, `Baiduspider`, `YandexBot`, `Sogou` - **AI crawlers**: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Bytespider`, `CCBot`, `anthropic-ai` - **Mo