pagination-and-crawlinglisted

# Pagination and Crawling Two failure modes dominate: missing a third of the records without noticing, and crawling forever because the site generates infinite URLs. Both are prevented by the same discipline — know how many records you expect, and bound the crawl explicitly. ## Identify the pagination type first | Type | Looks like | Reliability | |---|---|---| | **Cursor / token** | `?cursor=eyJpZCI6...` | Best. Stable under inserts | | **Keyset** | `?after_id=1042&limit=50` | Best. Same reason | | **Page number** | `?page=3` | Fine, but shifts if records are added mid-crawl | | **Offset** | `?offset=100&limit=50` | Same drift problem, plus slow on large tables | | **Link header / next URL** | `rel="next"` in headers or HTML | Follow it, don't construct URLs | | **Infinite scroll** | No URL change | Backed by an API. Find it | | **"Load more" button** | No URL change | Same | **Prefer cursor and keyset paging where offered.** Page and offset paging drift: if three records are inserted while you're on page 4, records shift across the boundary and you silently miss some and duplicate others. **Follow `rel="next"` rather than constructing URLs.** The site is telling you the next page; guessing the pattern breaks when the pattern changes. ```bash # The site's own next-page link, in headers or markup curl -sI 'https://api.example.com/items?page=1' | grep -i '^link:' curl -s https://example.com/listings | grep -oE '<link[^>]*rel="next"[^>]*>|<a[^>]*rel="next"[^>]*>' ``` ##