firecrawl-reliability-patterns

# Firecrawl Reliability Patterns ## Overview Production reliability patterns for Firecrawl scraping pipelines. Firecrawl's async crawl model, JS rendering, and credit-based pricing create specific reliability challenges: crawl jobs may time out, scraped content may be empty (bot detection, JS failures), and credits can be burned by runaway crawls. This skill covers battle-tested patterns for each. ## Instructions ### Step 1: Robust Crawl with Timeout and Backoff ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); async function reliableCrawl( url: string, opts: { limit: number; paths?: string[] }, timeoutMs = 600000 ) { const job = await firecrawl.asyncCrawlUrl(url, { limit: opts.limit, includePaths: opts.paths, scrapeOptions: { formats: ["markdown"], onlyMainContent: true }, }); const deadline = Date.now() + timeoutMs; let pollInterval = 2000; while (Date.now() < deadline) { const status = await firecrawl.checkCrawlStatus(job.id); if (status.status === "completed") return status; if (status.status === "failed") { throw new Error(`Crawl failed: ${status.error}`); } await new Promise(r => setTimeout(r, pollInterval)); pollInterval = Math.min(pollInterval * 1.5, 30000); // back off to 30s max } throw new Error(`Crawl timed out after ${timeoutMs}ms (job: ${job.id})`); } ``` ### Step 2: Content Quality Validation ...