agent-readylisted
Install: claude install-skill sitetiller/ai-visibility-skills
# Agent-ready: can machines read and use this site?
Goal: find what stops an AI crawler, a user-directed AI fetcher or a browsing agent from reaching and reading the content, in the order that matters: access first, then whether the text is in the HTML, then structure.
## Inputs
- A site URL and two or three representative pages (home, an article, a pricing or product page).
- What the user sees in a normal browser for those pages (headings, main text, prices, tables), to compare against what a crawler gets.
## Steps
1. **Access: robots.txt.** Run:
```bash
npx sitetiller crawlers example.com --json
npx sitetiller crawlers example.com/pricing --json
```
Read `bots[]` (`agent`, `purpose`, `allowed`, `rule`, `line`). Search and answer crawlers blocked by accident are the top fix; follow the `ai-crawlers` skill. Note which bots are user-initiated fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch (`ai-crawlers/references/sources.md`).
2. **Access: CDN and firewall.** robots.txt is not the only gate. Cloudflare's controls, effective 2026-09-15, are per domain: Search (crawling to build a search index), Training (to train or fine-tune models), and Agent ("user-directed agents visiting a page on behalf of a human", for example chat fetch bots and browser-use agents). Settings are Allow, Block on pages with ads, and Block; Training also has Disall