← ClaudeAtlas

robots-txt-and-llms-txtlisted

Audits and writes a site's robots.txt and llms.txt for search and AI crawlers, with a runnable offline checker that resolves which robots group every AI token actually obeys. Use when asked to check or fix robots.txt, AI crawler access (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot), whether AI bots are blocked, to create or review an llms.txt or llms-full.txt, or to fill the AI crawlability findings on an SEO or AEO audit slide.
savenjudesajen/seo-aeo-agent-skills-tools · ★ 0 · AI & Automation · score 73
Install: claude install-skill savenjudesajen/seo-aeo-agent-skills-tools
# robots.txt and llms.txt: audit and build Home page: [unilytics.ai/skills/robots-txt-and-llms-txt](https://unilytics.ai/skills/robots-txt-and-llms-txt/), the canonical source of this skill. Two small files, two very different weights. robots.txt decides whether search and answer engines can reach the site at all, so a mistake there is a visibility outage. llms.txt is a proposed, unconfirmed standard: cheap to ship, no measured effect yet. Audit robots.txt first and hardest; treat llms.txt as a low-cost hygiene item and say so to the client. ## Before you start (every run) 1. Fetch the live files yourself, never from memory or a crawler's cached copy: `/robots.txt` on every host variant (www and non-www, http and https, each subdomain), `/llms.txt`, `/llms-full.txt`, and the sitemap the robots file declares. Save each response body to a local file and record its status code and content type. A 200 that returns an HTML page is a soft 404, not an llms.txt. 2. Note the CMS or commerce platform (WordPress, Drupal, Webflow, HubSpot, Shopify, Adobe Commerce/Magento, BigCommerce, Salesforce Commerce Cloud). The platform decides which default rules should exist and which legacy rules are leftovers. 3. Check who actually serves the file. A CDN or edge rule can rewrite robots.txt, a host can serve a different file per variant, and an SEO plugin can generate (or overwrite) both robots.txt and llms.txt. Find the owner before you propose an edit, or the fix will