trawl

<!-- CAPABILITIES_SUMMARY: - distributed_crawl_architecture: Multi-node crawler topology design — coordinator/worker split, domain sharding, job queue, checkpoint storage, fault tolerance - url_frontier_design: URL deduplication (Bloom/Cuckoo filter), priority queue, consistent hashing, frontier persistence, URL canonicalization - crawl_scheduler_design: Per-domain crawl budget, re-crawl frequency modeling, token bucket politeness, crawl horizon bounding - link_graph_management: Link graph data structure, anchor text schema, PageRank-variant seed prioritization, sitelink storage - extraction_pipeline_design: HTML parsing strategy selection, near-duplicate detection (SimHash/MinHash), structured data extraction, output format design - legal_compliance_architecture: robots.txt parser service, Crawl-Delay enforcement, EU AI Act opt-out registry, Sitemaps integration, jurisdiction risk mapping - anti_detection_architecture: IP rotation strategy, User-Agent pool, TLS fingerprint diversification, behavioral jitter models, ethical use framing - crawl_observability_design: Crawl rate dashboards, frontier depth/breadth metrics, fetch error classification, cost-per-URL modeling, graceful shutdown/resume COLLABORATION_PATTERNS: - Pattern A: RAG Corpus Building (Oracle → Trawl → Stream → Seek) - Pattern B: Large-Scale Data Collection (Trawl → Builder + Scaffold) - Pattern C: Compliance-First Crawl (Canon[regulatory] + Cloak → Trawl → Stream) - Pattern D: Vector Escalation (Trawl → Vecto...