javascript-rendered-scrapinglisted

# JavaScript-Rendered Pages The usual sequence: a scraper returns nothing, the page looks fine in a browser, and the conclusion is "I need a headless browser". That conclusion is wrong more often than it's right. A headless browser is 10 to 50 times slower, far heavier on memory, much more fragile, and much more detectable. Exhaust the alternatives first. ## Check what you actually need ```bash # What the server sends, before any JavaScript runs curl -s https://example.com/page > raw.html wc -c raw.html grep -c "the-thing-you-want" raw.html ``` If your data is in `raw.html`, you're done — no browser needed. If it isn't, work down this list before reaching for one. **1. Hydration blobs.** Most modern frameworks embed the page's data as JSON in the initial HTML. ```bash # Next.js curl -s https://example.com | grep -oP '(?<=id="__NEXT_DATA__" type="application/json">).*?(?=</script>)' # Nuxt curl -s https://example.com | grep -oP 'window\.__NUXT__=.*?(?=</script>)' # Generic: any large JSON blob in a script tag curl -s https://example.com | grep -oP '(?s)<script[^>]*>\s*\{.*?\}\s*</script>' | head -c 3000 ``` This is the single highest-value check on this page. It's often the complete, typed record — better than anything you'd scrape from the rendered DOM. **2. The underlying API.** Open the page with the network tab filtered to XHR and fetch. The front end is calling something. That endpoint is usually paginated, returns clean JSON, and is far more stable than the ma