api-scrapinglisted

# api-scraping Most "scrape this website" problems are really "call the site's own API." A modern site's pages are shells that fetch JSON from an internal endpoint; that endpoint returns clean, structured, paginated data with no HTML to parse and no rendering to run. Reverse-engineer that call and you get the data faster, more reliably, and with far less code than a DOM scraper — the DOM scraper is the **fallback**, not the plan. (Adapted from Jerome Paulos's ["All the data can be yours"](https://jero.zone/posts/reverse-engineering-apis), turned into a loop an agent can run, plus HAR tooling and an anti-bot escalation ladder.) ## Gate first (every run) Before capturing anything, settle scope and permission — this is cheap and non-negotiable: - **What, how much, how often** — a one-off pull of a few thousand public rows is not a standing crawler hammering an endpoint. Size the job. - **`robots.txt` and Terms of Service** — fetch `https://<host>/robots.txt`; skim the ToS for an anti-scraping clause. `robots.txt` is advisory, not law, but ignoring an explicit disallow is a decision the **user** makes, not you. - **Stop and ask (`AskUserQuestion`) when** the data is behind a login you weren't given, the target is personal data (PII) at scale, the ToS explicitly forbids it, or the only way through is defeating a CAPTCHA/bot-challenge. Public data, respectful rate, no auth bypass is the green path. ## The loop 1. **Spot the API.** Content that appears *after* the