web-scrapinglisted

# Mass Scraping — Crawl4AI + Camoufox Verified 2026-08-27 (macOS arm64). Both engines and all three templates smoke-tested. ## Install (if `crawl4ai-python` is not on PATH) - **crawl4ai** — `github.com/unclecode/crawl4ai` · PyPI `crawl4ai` - **camoufox** — `github.com/daijro/camoufox` · PyPI `camoufox[geoip]` - **curl_cffi** — `github.com/lexiforest/curl_cffi` · PyPI `curl_cffi` Dedicated venv, then `crawl4ai-setup` and `python -m camoufox fetch`. Four traps, each verified here: 1. **Python 3.10–3.13 only.** 3.14 is not supported — if `python3` is 3.14, install 3.13 explicitly. 2. **`crawl4ai`, not `crawl4ai[all]`.** The extra pulls torch + transformers (GBs) that scraping never uses. 3. **`crawl4ai-setup`, not `playwright install chromium`.** Setup also installs **Patchright**, which rung 3 needs; without it the undetected adapter dies. 4. **Shim the venv python with a wrapper, never a symlink.** A symlinked venv python resolves `sys.prefix` to the system interpreter and cannot import crawl4ai: ```sh printf '#!/bin/sh\nexec "$VENV/bin/python" "$@"\n' > ~/.local/bin/crawl4ai-python ``` ## Entry points ```bash crwl https://example.com -o md -bc # one page → markdown crawl4ai-python script.py # anything bigger ``` Both are shims into the dedicated venv. **System `python3` cannot import crawl4ai/camoufox** — always `crawl4ai-python` or the venv binary directly. Don't pip-install these into another environment. `crwl`: `-o md` · `-o all` (JSON