robots-txt-and-llms-txtlisted
Install: claude install-skill savenjudesajen/seo-aeo-agent-skills-tools
# robots.txt and llms.txt: audit and build
Home page: [unilytics.ai/skills/robots-txt-and-llms-txt](https://unilytics.ai/skills/robots-txt-and-llms-txt/), the canonical source of this skill.
Two small files, two very different weights. robots.txt decides whether search and answer
engines can reach the site at all, so a mistake there is a visibility outage. llms.txt is a
proposed, unconfirmed standard: cheap to ship, no measured effect yet. Audit robots.txt
first and hardest; treat llms.txt as a low-cost hygiene item and say so to the client.
## Before you start (every run)
1. Fetch the live files yourself, never from memory or a crawler's cached copy:
`/robots.txt` on every host variant (www and non-www, http and https, each subdomain),
`/llms.txt`, `/llms-full.txt`, and the sitemap the robots file declares. Save each
response body to a local file and record its status code and content type. A 200 that
returns an HTML page is a soft 404, not an llms.txt.
2. Note the CMS or commerce platform (WordPress, Drupal, Webflow, HubSpot, Shopify, Adobe
Commerce/Magento, BigCommerce, Salesforce Commerce Cloud). The platform decides which default rules should exist and which legacy
rules are leftovers.
3. Check who actually serves the file. A CDN or edge rule can rewrite robots.txt, a host can
serve a different file per variant, and an SEO plugin can generate (or overwrite) both
robots.txt and llms.txt. Find the owner before you propose an edit, or the fix will