brightdata-reference-architecture

Featured

Implement Bright Data reference architecture with best-practice project layout. Use when designing new Bright Data integrations, reviewing project structure, or establishing architecture standards for Bright Data applications. Trigger with phrases like "brightdata architecture", "brightdata best practices", "brightdata project structure", "how to organize brightdata", "brightdata layout".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Bright Data Reference Architecture ## Overview Production-ready architecture for Bright Data scraping systems. Covers project layout, data pipeline design, and integration patterns for Web Unlocker, Scraping Browser, SERP API, and Datasets API. ## Prerequisites - Understanding of layered architecture - Node.js/TypeScript project setup - Database for storing scraped data ## Project Structure ``` my-scraper/ ├── src/ │ ├── brightdata/ │ │ ├── proxy.ts # Proxy config helper (zone, country, session) │ │ ├── client.ts # Axios client with proxy + retry │ │ ├── browser.ts # Scraping Browser connection manager │ │ ├── api.ts # REST API client (trigger, snapshot) │ │ ├── cache.ts # Response cache (LRU + optional Redis) │ │ └── types.ts # Shared TypeScript interfaces │ ├── scrapers/ │ │ ├── product-scraper.ts # Domain-specific scraper │ │ ├── serp-scraper.ts # Search result collector │ │ └── parser.ts # HTML → structured data (cheerio) │ ├── pipeline/ │ │ ├── scheduler.ts # Cron-based scraping scheduler │ │ ├── processor.ts # Raw HTML → clean data │ │ └── storage.ts # Database/file output │ ├── webhooks/ │ │ └── brightdata.ts # Webhook delivery handler │ └── api/ │ ├── health.ts # Health check endpoint │ └── scrape.ts # On-demand scrape endpoint ├── tests/ │ ├── unit/ ...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category