firecrawl-performance-tuning

# Firecrawl Performance Tuning ## Overview Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), `waitFor` tuning, `onlyMainContent`, and batch vs individual scraping. ## Latency Benchmarks | Operation | Typical | With JS Wait | With Screenshot | |-----------|---------|--------------|-----------------| | scrapeUrl (markdown) | 2-5s | 5-10s | 8-15s | | scrapeUrl (extract) | 3-8s | 8-15s | N/A | | crawlUrl (10 pages) | 20-40s | 40-80s | N/A | | mapUrl | 1-3s | N/A | N/A | | batchScrapeUrls (10) | 10-20s | 20-40s | N/A | ## Instructions ### Step 1: Minimize Formats (Biggest Win) ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); // SLOW: requesting everything const slow = await firecrawl.scrapeUrl(url, { formats: ["markdown", "html", "links", "screenshot"], // screenshot + full HTML = 3-5x slower }); // FAST: request only what you need const fast = await firecrawl.scrapeUrl(url, { formats: ["markdown"], // markdown only = fastest onlyMainContent: true, // skip nav/footer/sidebar }); ``` ### Step 2: Tune waitFor for JS-Heavy Pages ```typescript // Default: no JS wait (fastest, works for static sites) const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", { formats: ["markdown"], // N...