firecrawl-known-pitfalls

# Firecrawl Known Pitfalls ## Overview Real gotchas from production Firecrawl integrations. Each pitfall includes the bad pattern, why it fails, and the correct approach. Use this as a code review checklist. ## Pitfall 1: Unbounded Crawl (Credit Bomb) ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); // BAD: no limit — a docs site with 50K pages burns your entire credit balance await firecrawl.crawlUrl("https://docs.large-project.org"); // GOOD: always set limit, maxDepth, and path filters await firecrawl.crawlUrl("https://docs.large-project.org", { limit: 100, maxDepth: 3, includePaths: ["/api/*", "/guides/*"], excludePaths: ["/changelog/*", "/blog/*"], scrapeOptions: { formats: ["markdown"] }, }); ``` ## Pitfall 2: Not Specifying Output Format ```typescript // BAD: default format may not include markdown const result = await firecrawl.scrapeUrl("https://example.com"); console.log(result.markdown); // might be undefined! // GOOD: explicitly request the format you need const result = await firecrawl.scrapeUrl("https://example.com", { formats: ["markdown"], onlyMainContent: true, }); console.log(result.markdown); // guaranteed present ``` ## Pitfall 3: Not Waiting for JS-Heavy Pages ```typescript // BAD: SPAs show loading state, not content const result = await firecrawl.scrapeUrl("https://app.example.com/dashboard"); // result.markdown === "Loading..." or em...