change-detection-monitoringlisted
Install: claude install-skill manypicom/web-data-skills
# Change Detection and Monitoring
The naive way to keep a dataset current is to re-crawl everything on a schedule. It's expensive, it's rude to the target, and almost all of it is wasted — most pages haven't changed.
Change detection turns that into a delta: ask the source what changed, fetch only that, and diff to confirm. Done properly it cuts request volume by an order of magnitude and makes "what changed" a first-class output rather than something you infer.
## Ask before you fetch
Four mechanisms, cheapest first. Use them in this order.
**1. `lastmod` in the sitemap.** The site telling you which URLs changed and when. Free, and almost nobody uses it.
```bash
curl -s https://example.com/sitemap.xml \
| grep -oPz '(?s)<url>.*?</url>' | tr '\0' '\n' \
| grep -oP '(?<=<loc>)[^<]+|(?<=<lastmod>)[^<]+' \
| paste - - \
| awk -v since="2026-09-01" '$2 >= since {print $1}'
```
**2. Conditional requests.** `ETag` and `Last-Modified` let the server answer "unchanged" in a few hundred bytes.
```python
def fetch_if_changed(url, session, state):
prior = state.get(url, {})
headers = {}
if prior.get("etag"):
headers["If-None-Match"] = prior["etag"]
if prior.get("last_modified"):
headers["If-Modified-Since"] = prior["last_modified"]
r = session.get(url, headers=headers, timeout=30)
if r.status_code == 304:
return None # unchanged, cost us almost nothing
state[url] = {
"etag": r.headers.get(