web-fetchlisted

# Web Content Fetching Fetch web content in this order: 1. Prefer markdown-native endpoints (`content-type: text/markdown`) 2. Use selector-based HTML extraction for known sites 3. Use the bundled Bun fallback script when selectors fail ## Prerequisites Verify required tools before extracting: ```bash command -v curl >/dev/null || echo "curl is required" command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction" command -v bun >/dev/null || echo "bun is required for fetch.ts fallback" ``` Install Bun dependencies for the bundled script: ```bash cd ~/.claude/skills/web-fetch && bun install ``` ## Default Workflow Use this as the default flow for any URL: ```bash URL="<url>" CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)" if echo "$CONTENT_TYPE" | grep -q "markdown"; then curl -sL "$URL" else curl -sL "$URL" \ | html2markdown \ --include-selector "article,main,[role=main]" \ --exclude-selector "nav,header,footer,script,style" fi ``` ## Known Site Selectors | Site | Include Selector | Exclude Selector | |------|------------------|------------------| | platform.claude.com | `#content-container` | - | | docs.anthropic.com | `#content-container` | - | | developer.mozilla.org | `article` | - | | github.com (docs) | `article` | `nav,.sidebar` | | Generic | `article,main,[role=main]` | `nav,header,footer,script,style` | Example: ```bash curl -s