webreaper

# WebReaper — scraping & extraction CLI Three commands. Each is one shell call; output goes to stdout unless `--output`. ## What to run | The user wants… | Run | |---|---| | The readable text of one page | `webreaper scrape <url>` | | The readable text saved to a file | `webreaper scrape <url> --output page.md` | | Specific fields from one page | `webreaper scrape <url> --schema schema.json` | | Every page of a whole site (Markdown) | `webreaper crawl <url> > pages.jsonl` | | Every page of a whole site (fields) | `webreaper crawl <url> --schema schema.json` | | URLs on a site | `webreaper map <url>` | | URLs matching a substring | `webreaper map <url> --search /blog/ --max-urls 50` | | A JS-rendered page (SPA) | `webreaper scrape <url> --browser` | | A bot-protected site | `webreaper scrape <url> --browser --auto-stealth` | | Fields from every linked page | `webreaper scrape <index-url> --follow "<css selector>" --schema schema.json` | ## Whole-site crawl in one command To cover an entire site, use `crawl`: it recursively follows every on-domain link from the start URL, extracts each page, and streams JSON Lines (one object per page). Markdown by default; `--schema` switches to field extraction. ```bash # Every page, as Markdown, to a file. webreaper crawl https://example.com > pages.jsonl # Every page, specific fields, bounded. webreaper crawl https://example.com --schema schema.json --max-pages 200 ``` Flags: `--max-pages <n>` (default 1000), `--max-depth <n>` (hops...