scraper-maintenancelisted

# Scraper Maintenance Every scraper you write is a dependency on someone else's markup, and they will change it without telling you. The question is not whether it breaks but whether you find out before or after the data is used. Most teams find out after. The fix is monitoring the *shape* of the output rather than whether the job exited zero. ## The three ways scrapers fail | Failure | Symptom | How you find out | |---|---|---| | **Hard break** | Exception, zero rows, HTTP error | Immediately. The easy case | | **Partial break** | One selector stops matching, rest still work | Never, without per-field monitoring | | **Semantic drift** | Selector still matches, now the wrong element | Never, without manual checks | **The second and third are the dangerous ones**, and neither raises an exception. A scraper that returns 800 of 1,000 records with an empty `email` field looks like a successful run in every log. Semantic drift is the worst of the three: a site restructures, the selector still matches something, and now you're extracting the "related products" price instead of the product price. Every value is plausible. Every value is wrong. ## Monitor the output shape, not the exit code ```python import json, pathlib, datetime as dt HIST = pathlib.Path("runs") def record_run(name, rows, extractor_version): n = max(len(rows), 1) fields = {k for r in rows for k in r} profile = { "at": dt.datetime.now(dt.timezone.utc).isoformat(), "version": ext