web-scraper

# Web Scraper ## Overview Extract content and data from websites using various techniques including crawling, scraping, and structured data extraction. ## When to Use - Extract text content from web pages - Crawl entire websites - Collect structured data - Research and gather information - Monitor website changes - Extract tables and lists ## Tools Available ### Content Extraction ```javascript // Use extract_content_from_websites for structured extraction // Supports batch processing of multiple URLs // Returns JSON format with extracted content ``` ### Task Format ```javascript { tasks: [ { url: "https://example.com", prompt: "Extract specific information", task_name: "optional_name" } ] } ``` ## Usage Patterns ### Simple Content Extraction ```javascript // Extract main content from a page const result = await extract_content_from_websites({ tasks: [{ url: "https://news.example.com/article", prompt: "Extract the title, author, date, and main content" }] }); ``` ### Batch URL Processing ```javascript // Process multiple URLs in parallel const urls = [ "https://site.com/page1", "https://site.com/page2", "https://site.com/page3" ]; const results = await extract_content_from_websites({ tasks: urls.map((url, i) => ({ url, prompt: "Extract all product information, prices, and descriptions", task_name: `product_${i}` })) }); ``` ### Data Mining ...