ai-logslisted
Install: claude install-skill sitetiller/ai-visibility-skills
# AI crawlers in your logs
robots.txt says who may visit. Logs say who did. Anyone can put "GPTBot" in a user agent, so a hit only counts as OpenAI's when the IP is inside OpenAI's published ranges. This skill reports both.
## Inputs
- One or more access logs in common or combined log format (nginx and Apache defaults). Gzip: decompress first.
- Other formats (Cloudflare Logpush JSON, CDN CSV): convert to combined format, or ask for the columns ip, time, request, status, referer, user agent and convert with a short script.
## Steps
1. Run the analysis (downloads the vendors' IP lists once per run):
```bash
npx sitetiller logs access.log access.log.1 --json > ai-logs.json
```
Offline: `--no-verify` counts by user agent only and marks every hit unverifiable.
2. Read per bot: `hits`, `verified`, `spoofed`, `unverifiable`, `ips`, `status` mix and top `paths`.
3. Interpret:
| Finding | Meaning | Action |
|---|---|---|
| `verified` hits on search bots (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot) | The engines can read the site | Check that top paths are the pages you want cited |
| Search bot hits mostly `4xx/5xx` or `403` | Bot reaches the server but is refused | Check WAF/CDN rules and robots.txt (`ai-crawlers` skill) |
| `spoofed` > 0 | Scrapers using a vendor's name | Safe to rate-limit or block those IPs; it does not affect the real bot |
| `unverifiable` | Owner publishes no IP list (Anthropic, Apple, Amazon, Meta, others) | Treat counts as an upper boun