firecrawl-performance-tuning

Featured

Optimize Firecrawl scraping performance with caching, batch scraping, and format selection. Use when experiencing slow scrapes, optimizing credit usage per page, or building high-throughput scraping pipelines. Trigger with phrases like "firecrawl performance", "optimize firecrawl", "firecrawl latency", "firecrawl caching", "firecrawl slow", "firecrawl batch".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Firecrawl Performance Tuning ## Overview Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), `waitFor` tuning, `onlyMainContent`, and batch vs individual scraping. ## Latency Benchmarks | Operation | Typical | With JS Wait | With Screenshot | |-----------|---------|--------------|-----------------| | scrapeUrl (markdown) | 2-5s | 5-10s | 8-15s | | scrapeUrl (extract) | 3-8s | 8-15s | N/A | | crawlUrl (10 pages) | 20-40s | 40-80s | N/A | | mapUrl | 1-3s | N/A | N/A | | batchScrapeUrls (10) | 10-20s | 20-40s | N/A | ## Instructions ### Step 1: Minimize Formats (Biggest Win) ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); // SLOW: requesting everything const slow = await firecrawl.scrapeUrl(url, { formats: ["markdown", "html", "links", "screenshot"], // screenshot + full HTML = 3-5x slower }); // FAST: request only what you need const fast = await firecrawl.scrapeUrl(url, { formats: ["markdown"], // markdown only = fastest onlyMainContent: true, // skip nav/footer/sidebar }); ``` ### Step 2: Tune waitFor for JS-Heavy Pages ```typescript // Default: no JS wait (fastest, works for static sites) const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", { formats: ["markdown"], // N...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Featured

firecrawl-cost-tuning

Optimize Firecrawl costs through crawl limits, format selection, caching, and credit monitoring. Use when analyzing Firecrawl billing, reducing API costs, or implementing credit budget alerts. Trigger with phrases like "firecrawl cost", "firecrawl billing", "reduce firecrawl costs", "firecrawl pricing", "firecrawl credits", "firecrawl budget".

2,926 Updated yesterday
jeremylongshore
Data & Documents Listed

firecrawl

Firecrawl produces cleaner markdown than WebFetch, handles JavaScript-heavy pages, and avoids content truncation. This skill should be used when fetching URLs, scraping web pages, converting URLs to markdown, extracting web content, searching the web, crawling sites, mapping URLs, LLM-powered extraction, autonomous data gathering with the Agent API, interacting with scraped pages (clicking, filling forms, extracting dynamic content via Interact API), or fetching AI-generated documentation for GitHub repos via DeepWiki. Provides complete coverage of Firecrawl v2 API endpoints including parallel agents, spark-1-fast model, sitemap-only crawling, and the Interact API for post-scrape browser interaction.

41 Updated 1 weeks ago
tdimino
AI & Automation Featured

firecrawl-architecture-variants

Choose and implement Firecrawl architecture patterns for different scales and use cases. Use when designing new Firecrawl integrations, choosing between on-demand/scheduled/pipeline architectures, or planning scraping infrastructure. Trigger with phrases like "firecrawl architecture", "firecrawl blueprint", "how to structure firecrawl", "firecrawl at scale", "firecrawl pipeline design".

2,926 Updated yesterday
jeremylongshore