ClaudeAtlasBot

ClaudeAtlasBot/1.0 (+https://claudeatlas.com/bot)

If you found this page from a User-Agent string in your server logs: that was us, and this page explains exactly what we did, why, and how to opt out.

Who we are

ClaudeAtlas is an independently maintained, open-source index of Claude ecosystem skills, plugins and MCP servers. Most of our own traffic is automated — crawlers, SEO tools and AI agents — so we run a small research crawler to measure the reciprocal question: how does the public web treat a polite, self-declared bot? To be precise about what we are: ClaudeAtlasBot is a plain crawler with a fixed probe list — no language model decides what it fetches. Servers can't see whether a model sits behind a request; they see the declaration, and that is what we measure. The rationale and scope are public: docs/reciprocal-agent-pass-scope.md.

What the bot does

Read-only, meta-only. It records status codes, a few response headers, robots.txt directives, response sizes and whether structured data is present. It does not keep page content for reuse and does not collect personal data.
GET only. No POST, no forms, no logins, nothing behind authentication.
A fixed probe set. The exact list is below. It does not crawl beyond it — no link following, no site walk.
At most one request every 3 seconds per host (redirect hops included), at most 6 hosts in flight at a time, a 15-second timeout per request, at most 5 redirects, and at most 512 KB read per response.
Fetches and honors robots.txt first — both a User-agent: ClaudeAtlasBot group and User-agent: *. A path you disallow is never requested. If robots.txt is unreachable (5xx / timeout) we treat the whole site as disallowed.
Never retries a 4xx or 5xx. A single retry only on a dropped connection.
Single pass, run manually. Not continuous. The current target list is about thirty domains — the operators whose bots appear in our own logs.

How to identify it

The exact User-Agent string on every request:

ClaudeAtlasBot/1.0 (+https://claudeatlas.com/bot)

In this first version requests are not cryptographically signed, so anyone can spoof the string above. A later version will sign every request per RFC 9421 / Web Bot Auth, with a public key directory at https://claudeatlas.com/.well-known/http-message-signatures-directory — the same mechanism ClaudeAtlas already verifies on inbound traffic. Until then, a request claiming to be ClaudeAtlasBot that ignores the limits above is not us.

What we fetch

Per site, in this order, each step gated by robots.txt:

/robots.txt
/ — three times: with our User-Agent and an HTML Accept; with our User-Agent and Accept: text/markdown, text/html;q=0.9 (does the site offer a markdown rendition to agents?); and once with a standard desktop-browser User-Agent for a lightweight cloaking comparison — status and size only.
/llms.txt
/llms-full.txt
/.well-known/agents.json
/.well-known/mcp/server-card.json
/.well-known/http-message-signatures-directory
/ai.txt

That is 10 requests per site at most, spread over roughly 27 seconds.

What we record — and what we don't

Recorded

HTTP status codes
Headers: server, content-type, x-robots-tag, content-signal, cf-mitigated, presence of cf-ray
Byte counts and redirect counts
robots.txt directives (which bots are named, what is disallowed, Sitemap, Content-Signal)
Number of JSON-LD blocks on the homepage
Whether a markdown rendition is offered
Whether each well-known path exists and looks plausible

Not recorded

Page text or HTML
Images, scripts, stylesheets (never requested)
Forms, cookies, session state
Anything behind authentication
Any personal data

How to block it

Add this to your robots.txt:

User-agent: ClaudeAtlasBot
Disallow: /

A User-agent: * disallow is honored too.
It takes effect on the next pass — robots.txt is the first thing we fetch, every time.
A blocked or challenged response is itself recorded as data. We never work around it, never switch User-Agents to get past it, and never retry it.
Prefer not to appear at all? Ask (below) and we will add your domain to an exclusion list.

The resulting dataset

Every pass writes one small JSON file and one report, both public: data/reciprocal-pass.json and docs/reciprocal-pass-report.md. Methodology in one sentence: a fixed target list of operators whose bots appear in our own request log, a fixed probe set, one pass, and aggregate block / challenge / toll rates alongside robots.txt posture, markdown negotiation and agent-web standards adoption. Being blocked is a data point, not a complaint.

Contact

Open an issue at github.com/dwalshx/ClaudeAtlas/issues and mention ClaudeAtlasBot in the title. We respond within a few days and will add your domain to an exclusion list on request.

Source for the crawler: scripts/reciprocal-pass.js (scheduling) and scripts/lib/reciprocal-probe.js (the policy constants this page is rendered from).