firecrawl-architecture-variants

# Firecrawl Architecture Variants ## Overview Three deployment architectures for Firecrawl at different scales: on-demand scraping for simple use cases, scheduled crawl pipelines for content monitoring, and real-time ingestion pipelines for AI/RAG applications. Choose based on volume, latency requirements, and cost budget. ## Decision Matrix | Factor | On-Demand | Scheduled Pipeline | Real-Time Pipeline | |--------|-----------|-------------------|-------------------| | Volume | < 500/day | 500-10K/day | 10K+/day | | Latency | Sync (2-10s) | Async (hours) | Async (minutes) | | Use Case | Single page lookup | Site monitoring | Knowledge base, RAG | | Credit Control | Per-request | Per-crawl budget | Credit pipeline | | Complexity | Low | Medium | High | ## Instructions ### Architecture 1: On-Demand Scraping ``` User Request → Backend API → firecrawl.scrapeUrl → Clean Content → Response ``` Best for: chatbots, content preview, single-page extraction. ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); // Simple API endpoint app.post("/api/scrape", async (req, res) => { const { url } = req.body; const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"], onlyMainContent: true, waitFor: 3000, }); res.json({ title: result.metadata?.title, content: result.markdown, url: result.metadata?.sourceURL, }); }); // With LLM extraction app....