crawllisted

# Web Crawling Skill ## Overview This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis. ## When to Use This Skill Use this skill when users request: - Crawling or spidering a URL or domain - Subdomain discovery and enumeration - Endpoint and API discovery - JavaScript file extraction and analysis - Web application reconnaissance or mapping - Bug bounty scoping and enumeration - Attack surface discovery **Trigger keywords:** crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance ## Quick Start The primary tool is `scripts/crawl.py`, which provides a unified interface to both hakrawler and gospider: ```bash # Basic crawl - creates crawl_example.com_TIMESTAMP/ python scripts/crawl.py https://example.com # Deep crawl with subdomain discovery python scripts/crawl.py https://example.com --depth 3 --subdomains # Crawl with JS analysis python scripts/crawl.py https://example.com --js-analysis # Use only one tool python scripts/crawl.py https://example.com --tool hakrawler # Custom output directory and timeout python scripts/crawl.py https://example.com --output-dir ./results --timeout 600 ``` **