scrape

Solid

Scrape any webpage as clean markdown via Bright Data Web Unlocker API. Bypasses bot detection and CAPTCHA. Requires BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE environment variables.

Web & Frontend 33,961 stars 4034 forks Updated today MIT

Install

View on GitHub

Quality Score: 91/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
67
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Bright Data - Web Scraper Scrape any webpage and get clean markdown content using Bright Data's Web Unlocker API. Automatically bypasses bot detection and CAPTCHA. ## Setup **1. Get your API Key:** Get a key from [Bright Data Dashboard](https://brightdata.com/cp). **2. Create a Web Unlocker zone:** Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone". **3. Set environment variables:** ```bash export BRIGHTDATA_API_KEY="your-api-key" export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name" ``` ## Usage ```bash bash scripts/scrape.sh "url" ``` **Parameters:** - `url` (required): The webpage URL to scrape **Examples:** ```bash # Scrape a news article bash scripts/scrape.sh "https://example.com/article" # Scrape a product page bash scripts/scrape.sh "https://shop.example.com/product/123" ``` ## Output Format Returns clean markdown content extracted from the webpage: ```markdown # Page Title Main content of the page converted to markdown format... ## Section Heading More content... ``` ## Features - **Bot Detection Bypass**: Automatically handles anti-bot measures - **CAPTCHA Solving**: Bypasses CAPTCHA challenges - **Clean Markdown**: Returns well-formatted markdown content - **JavaScript Rendering**: Handles JavaScript-heavy pages ## Dependencies - `curl` - For API requests

Details

Author
davila7
Repository
davila7/claude-code-templates
Created
1 years ago
Last Updated
today
Language
Python
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category