firecrawl-architecture-variants

Featured

Choose and implement Firecrawl architecture patterns for different scales and use cases. Use when designing new Firecrawl integrations, choosing between on-demand/scheduled/pipeline architectures, or planning scraping infrastructure. Trigger with phrases like "firecrawl architecture", "firecrawl blueprint", "how to structure firecrawl", "firecrawl at scale", "firecrawl pipeline design".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Firecrawl Architecture Variants ## Overview Three deployment architectures for Firecrawl at different scales: on-demand scraping for simple use cases, scheduled crawl pipelines for content monitoring, and real-time ingestion pipelines for AI/RAG applications. Choose based on volume, latency requirements, and cost budget. ## Decision Matrix | Factor | On-Demand | Scheduled Pipeline | Real-Time Pipeline | |--------|-----------|-------------------|-------------------| | Volume | < 500/day | 500-10K/day | 10K+/day | | Latency | Sync (2-10s) | Async (hours) | Async (minutes) | | Use Case | Single page lookup | Site monitoring | Knowledge base, RAG | | Credit Control | Per-request | Per-crawl budget | Credit pipeline | | Complexity | Low | Medium | High | ## Instructions ### Architecture 1: On-Demand Scraping ``` User Request → Backend API → firecrawl.scrapeUrl → Clean Content → Response ``` Best for: chatbots, content preview, single-page extraction. ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); // Simple API endpoint app.post("/api/scrape", async (req, res) => { const { url } = req.body; const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"], onlyMainContent: true, waitFor: 3000, }); res.json({ title: result.metadata?.title, content: result.markdown, url: result.metadata?.sourceURL, }); }); // With LLM extraction app....

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category