← ClaudeAtlas

fix-broken-scraperlisted

Diagnoses why a working web scraper or crawler broke and repairs it one verified step at a time — separates stale CSS selectors and site redesigns from real blocking, 403/429 responses, expired sessions, changed URL routing, and content that moved into embedded JSON or client-side rendering. Use this whenever a scraper, spider, crawler or parser that used to work now returns empty results, None, an empty list, zero rows, HTTP 403/429, a CAPTCHA page or wrong data — whenever someone says their scraping script stopped working, broke overnight, worked yesterday, is suddenly blocked, or their BeautifulSoup, Scrapy, Playwright, Puppeteer or requests code returns nothing.
prashant-cr/skills · ★ 1 · Data & Documents · score 75
Install: claude install-skill prashant-cr/skills
# Fix a broken scraper Scrapers break constantly, and almost all of the cost is misdiagnosis. "I'm being blocked" is the loudest hypothesis, so people reach for headless browsers, proxies and spoofed headers — while the page returns HTTP 200 and the selectors simply no longer match. Each wrong guess is permanent: a browser added for a markup change stays in the stack forever, slower and more fragile than the code it replaced, and it never addressed the fault. So this skill classifies before it repairs, and changes one thing at a time. ## Workflow ### 1. Get the symptom precisely Vague symptoms produce vague fixes. Pin down: - **What exactly comes back?** Empty list, `None`, a partial row, wrong values, an exception, a non-200 status — these are different failures with different causes. - **All URLs or some?** A subset failing points at page variants, not at blocking. - **Since when, and what changed?** A site redesign, a dependency upgrade, a new deployment, or a changed IP each suggest a different class. "It worked yesterday" is worth pinning to a date. - **First request or after N?** Failing immediately is a different problem from failing after two hundred requests, which is pacing or session state. Ask when these aren't given. The answers routinely make the diagnosis obvious before any request. ### 2. Reproduce at the smallest scale Get to a single failing URL and a single extraction step. A scraper that fails inside a Scrapy pipeline, a retry wrapper and a