crawllisted

# /crawl — Tiered Web Page Fetcher Fetch web pages that may be JS-rendered or bot-protected, and return clean markdown (or raw HTML). It has two tiers: a **direct HTTP fetch** that needs zero configuration, and **Cloudflare Browser Rendering** — a managed headless Chromium that renders JavaScript, follows redirects, and bypasses most WAFs — which activates only when you supply your own Cloudflare credentials. The bundled script is `scripts/crawl.ts` (run with `npx tsx`). ## Requirements **Out of the box (no setup):** the direct-fetch tier needs only Node.js and `tsx` (`npm install -g tsx`). It handles static HTML, many simple pages, and `.md`/`.md.txt`/`.txt` endpoints. JS-rendered SPAs and bot-protected pages will fail this tier — that's expected, and where the Cloudflare tier comes in. **Optional — bring your own Cloudflare Browser Rendering (recommended for JS/SPA/blocked pages):** Cloudflare's Browser Rendering REST API runs a real headless Chromium in their network. It's a per-account API — there's no shared endpoint, so you supply **your own** account ID and token via environment variables: | Variable | Required for | What it is | |---|---|---| | `CLOUDFLARE_ACCOUNT_ID` | CF tier | Your Cloudflare account ID (32-char hex) | | `CLOUDFLARE_BR_TOKEN` | CF tier | An API token scoped to **Account → Browser Rendering → Edit** | Setup: 1. Sign in at `https://dash.cloudflare.com` and copy your **Account ID** (right sidebar). 2. Create an API token at `https://dash.cloudf