← ClaudeAtlas

scrape-feasibility-auditlisted

Audits a public website to determine how hard it will be to scrape and what to build it with — identifies bot detection (Cloudflare, Akamai, DataDome, HUMAN/PerimeterX, Kasada, Imperva, AWS WAF), CAPTCHA types (reCAPTCHA, hCaptcha, Turnstile, Arkose, GeeTest), robots.txt rules, rate limits, and whether content is server- or client-rendered, then recommends open-source libraries. Use this whenever the user asks whether a site can be scraped, why their scraper is getting blocked or returning empty pages, what anti-bot protection a site uses, which scraping library or framework to pick, or is planning any data-collection or crawling project — even if they only mention a URL and extracting data from it.
prashant-cr/skills · ★ 1 · AI & Automation · score 75
Install: claude install-skill prashant-cr/skills
# Scrape feasibility audit Answers two questions before anyone writes a scraper: **should we collect this, and what will it take?** Getting these wrong is expensive in a specific way — teams build a Playwright cluster for a site that publishes JSON-LD, or spend a week fighting Kasada for data behind an official API that would have taken an afternoon. The output is a decision, with evidence, not a pile of observations. ## Scope Covers **public, unauthenticated content**: pages any visitor can reach without logging in. Outside scope, and worth saying plainly to the user if a request lands here: - Content behind authentication, paywalls, or purchased access - Defeating an interactive CAPTCHA that gates the content itself - Evading a block on a site that has explicitly refused automated access These aren't arbitrary lines. Each marks the point where a technical obstacle has become a stated decision by the operator, and where routing around it stops being an engineering question. When an audit lands there, the useful contribution is to say so and redirect to a sanctioned path — that is a more valuable answer than a fragile workaround, because it is the one that still works in six months. ## Workflow ### 1. Look for the sanctioned path first Do this before probing anything. It is the step most often skipped and the one that most often ends the project early, in a good way. Check for: - An official API — try `/api`, `/api/docs`, `/openapi.json`, `/graphql`, and a web sear