← ClaudeAtlas

llms-visibilitylisted

Make websites, docs, and blogs readable to LLMs and AI agents: llms.txt, .md routes, Accept negotiation, Content-Signal
svyatov/agent-toolkit · ★ 0 · AI & Automation · score 75
Install: claude install-skill svyatov/agent-toolkit
# Make a website visible to LLMs Apply the techniques below to get a site's content to LLMs and AI agents in clean Markdown over standard HTTP. Companion to <https://evilmartians.com/chronicles/how-to-make-your-website-visible-to-llms.md>. Most of these are emerging conventions, not committed standards. No major provider has formally promised to read `llms.txt` or `.md` routes. Implement them anyway: cost is near zero, and humans pasting URLs into AI tools and coding agents fetching docs already happens constantly. ## Workflow Each step is independently shippable. ### 1. Audit `robots.txt` and add `Content-Signal:` Confirm the site isn't blocking AI crawlers (`GPTBot`, `ClaudeBot`, `PerplexityBot`). Then add Cloudflare's directive (CC0): ``` User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes ``` Three orthogonal signals: `search` (search results), `ai-input` (live AI context), `ai-train` (model training). Adjust to the owner's policy. Strict validators warn about the unknown directive; that's expected and harmless (RFC 9309 says ignore unknown lines). ### 2. Ship `/llms.txt` A static Markdown file at the site root. Format from llmstxt.org: ```markdown # Site Name > One-sentence description of what this site is and who it serves. ## Documentation - [Quick Start](/docs/start): Get up and running in 5 minutes - [API Reference](/docs/api): Full endpoint documentation ``` Curate. It's a README for AI-mediated conversations, not a sitemap. ### 3. S