← ClaudeAtlas

agent-readylisted

Check whether AI assistants and agents can actually read and use a site: crawler access (robots.txt and CDN controls such as Cloudflare's Search, Training and Agent settings), whether the content is in the server's HTML or only appears after JavaScript runs, clean HTML structure and real links, stable URLs, schema, and the optional llms.txt. Marks emerging standards such as WebMCP as emerging. Use when the user asks 'is my site agent-ready', 'can AI agents read my site', 'my content loads with JavaScript, will ChatGPT see it', 'is my site readable by AI', 'agentic browsing', or 'WebMCP'. Every claim is sourced or labelled as unverified.
sitetiller/ai-visibility-skills · ★ 0 · AI & Automation · score 75
Install: claude install-skill sitetiller/ai-visibility-skills
# Agent-ready: can machines read and use this site? Goal: find what stops an AI crawler, a user-directed AI fetcher or a browsing agent from reaching and reading the content, in the order that matters: access first, then whether the text is in the HTML, then structure. ## Inputs - A site URL and two or three representative pages (home, an article, a pricing or product page). - What the user sees in a normal browser for those pages (headings, main text, prices, tables), to compare against what a crawler gets. ## Steps 1. **Access: robots.txt.** Run: ```bash npx sitetiller crawlers example.com --json npx sitetiller crawlers example.com/pricing --json ``` Read `bots[]` (`agent`, `purpose`, `allowed`, `rule`, `line`). Search and answer crawlers blocked by accident are the top fix; follow the `ai-crawlers` skill. Note which bots are user-initiated fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch (`ai-crawlers/references/sources.md`). 2. **Access: CDN and firewall.** robots.txt is not the only gate. Cloudflare's controls, effective 2026-09-15, are per domain: Search (crawling to build a search index), Training (to train or fine-tune models), and Agent ("user-directed agents visiting a page on behalf of a human", for example chat fetch bots and browser-use agents). Settings are Allow, Block on pages with ads, and Block; Training also has Disall