firecrawl-load-scale

# Firecrawl Load & Scale ## Overview Load test and scale Firecrawl scraping pipelines. Firecrawl's rate limits are per-plan (RPM and concurrent connections), so scaling means maximizing throughput within those limits using batch scraping, async crawls, and queue-based request management. ## Rate Limits by Plan | Plan | Scrape RPM | Concurrent Crawls | Max Batch Size | |------|-----------|-------------------|----------------| | Free | 10 | 2 | 10 | | Hobby | 20 | 3 | 50 | | Standard | 50 | 5 | 100 | | Growth | 100 | 10 | 100 | | Scale | 500+ | 50+ | 100 | ## Instructions ### Step 1: Measure Baseline Throughput ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); async function measureThroughput(urls: string[], concurrency: number) { const start = Date.now(); const results: Array<{ url: string; durationMs: number; success: boolean; chars: number }> = []; // Process in batches of `concurrency` for (let i = 0; i < urls.length; i += concurrency) { const batch = urls.slice(i, i + concurrency); const batchResults = await Promise.all( batch.map(async url => { const t0 = Date.now(); try { const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] }); return { url, durationMs: Date.now() - t0, success: true, chars: result.markdown?.length || 0 }; } catch { return { url, durationMs: Date.now() - t0, ...