harvest-deep-crawl

Featured

Multi-page deep crawling - documentation sites, wikis, knowledge bases

AI & Automation 552 stars 45 forks Updated 6 days ago MIT

Install

View on GitHub

Quality Score: 94/100

Stars 20%
91
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Harvest Deep Crawl Crawl multi-page websites following internal links to a specified depth. Ideal for building complete knowledge bases from documentation sites, wikis, and reference materials. ## Usage ``` /crawl <url> --depth <N> ``` ## Examples ```bash # Crawl docs site 3 levels deep /crawl https://docs.example.com --depth 3 # Crawl a specific section /crawl https://docs.example.com/api --depth 2 # Crawl with page limit /crawl https://wiki.example.com --depth 5 --max-pages 50 ``` ## Parameters | Param | Default | Description | |-------|---------|-------------| | `--depth` | 2 | Max link-following depth | | `--max-pages` | 100 | Max pages to crawl | | `--same-domain` | true | Stay on same domain | | `--include` | * | URL pattern to include | | `--exclude` | - | URL pattern to exclude | ## How It Works 1. Start at root URL, extract all internal links 2. Follow links up to specified depth (BFS order) 3. Extract content from each page 4. Deduplicate pages with > 90% content overlap 5. Build table of contents from page hierarchy 6. Merge into coherent knowledge base 7. Save to `.claude/cache/agents/harvest/crawl-{domain}/` ## Output Structure ``` crawl-{domain}-{timestamp}/ index.md # Table of contents + summary page-001.md # First page content page-002.md # Second page content ... metadata.json # Crawl stats, URLs, timings ``` ## Crawl Engine ### Primary: crawl4ai (Docker port 11235) ```bash curl -s http://localhost:11235/cr...

Details

Author
vibeeval
Repository
vibeeval/vibecosystem
Created
8 months ago
Last Updated
6 days ago
Language
C#
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

harvest-single

Single page smart extraction - articles, docs, blog posts to clean markdown

552 Updated 6 days ago
vibeeval
AI & Automation Solid

webcrawler-deep-crawl

Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation header footer, website to markdown, website to text, multi-page extraction, bulk page scraping, clean markdown from URL, docs site to markdown corpus, site to clean corpus. Also applies to building RAG pipelines, indexing a customer site, syncing docs into a vector store, generating training corpora from any docs hub, or expanding a single start URL into a clean corpus of every reachable in-scope page.

6,374 Updated 2 weeks ago
browser-act
AI & Automation Listed

web-scraping

Mass crawl/scrape sites incl. anti-bot (Cloudflare, Akamai, DataDome, PerimeterX). Crawl4AI = scale (arun_many, dispatchers, deep crawl, LLM-ready markdown); Camoufox = fingerprint evasion. For: scrape N pages, crawl a site, extract structured data, 403/challenge/bot-detection, captcha.

3 Updated 2 months ago
sergeyizmailov