firecrawl-reliability-patterns

Featured

Implement Firecrawl reliability patterns: circuit breakers, crawl fallbacks, and content validation. Use when building fault-tolerant scraping pipelines, implementing crawl-to-scrape fallback, or adding content quality gates to Firecrawl integrations. Trigger with phrases like "firecrawl reliability", "firecrawl circuit breaker", "firecrawl fallback", "firecrawl resilience", "firecrawl fault tolerant".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Firecrawl Reliability Patterns ## Overview Production reliability patterns for Firecrawl scraping pipelines. Firecrawl's async crawl model, JS rendering, and credit-based pricing create specific reliability challenges: crawl jobs may time out, scraped content may be empty (bot detection, JS failures), and credits can be burned by runaway crawls. This skill covers battle-tested patterns for each. ## Instructions ### Step 1: Robust Crawl with Timeout and Backoff ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); async function reliableCrawl( url: string, opts: { limit: number; paths?: string[] }, timeoutMs = 600000 ) { const job = await firecrawl.asyncCrawlUrl(url, { limit: opts.limit, includePaths: opts.paths, scrapeOptions: { formats: ["markdown"], onlyMainContent: true }, }); const deadline = Date.now() + timeoutMs; let pollInterval = 2000; while (Date.now() < deadline) { const status = await firecrawl.checkCrawlStatus(job.id); if (status.status === "completed") return status; if (status.status === "failed") { throw new Error(`Crawl failed: ${status.error}`); } await new Promise(r => setTimeout(r, pollInterval)); pollInterval = Math.min(pollInterval * 1.5, 30000); // back off to 30s max } throw new Error(`Crawl timed out after ${timeoutMs}ms (job: ${job.id})`); } ``` ### Step 2: Content Quality Validation ...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category