brightdata-reference-architecture

# Bright Data Reference Architecture ## Overview Production-ready architecture for Bright Data scraping systems. Covers project layout, data pipeline design, and integration patterns for Web Unlocker, Scraping Browser, SERP API, and Datasets API. ## Prerequisites - Understanding of layered architecture - Node.js/TypeScript project setup - Database for storing scraped data ## Project Structure ``` my-scraper/ ├── src/ │ ├── brightdata/ │ │ ├── proxy.ts # Proxy config helper (zone, country, session) │ │ ├── client.ts # Axios client with proxy + retry │ │ ├── browser.ts # Scraping Browser connection manager │ │ ├── api.ts # REST API client (trigger, snapshot) │ │ ├── cache.ts # Response cache (LRU + optional Redis) │ │ └── types.ts # Shared TypeScript interfaces │ ├── scrapers/ │ │ ├── product-scraper.ts # Domain-specific scraper │ │ ├── serp-scraper.ts # Search result collector │ │ └── parser.ts # HTML → structured data (cheerio) │ ├── pipeline/ │ │ ├── scheduler.ts # Cron-based scraping scheduler │ │ ├── processor.ts # Raw HTML → clean data │ │ └── storage.ts # Database/file output │ ├── webhooks/ │ │ └── brightdata.ts # Webhook delivery handler │ └── api/ │ ├── health.ts # Health check endpoint │ └── scrape.ts # On-demand scrape endpoint ├── tests/ │ ├── unit/ ...