firecrawl-observability

# Firecrawl Observability ## Overview Monitor Firecrawl web scraping pipelines for success rates, credit consumption, content quality, and latency. Key signals: scrape success rate, crawl job completion, credit burn velocity, extraction quality (did markdown actually contain useful content vs error pages), and webhook delivery health. ## Key Metrics | Metric | Type | Why It Matters | |--------|------|---------------| | `firecrawl_scrapes_total` | Counter | Track scrape volume and success rate | | `firecrawl_credits_used` | Counter | Monitor credit consumption | | `firecrawl_scrape_duration_ms` | Histogram | Detect latency issues | | `firecrawl_content_quality` | Counter | Catch empty/error pages | | `firecrawl_crawl_jobs_total` | Counter | Track crawl job outcomes | ## Instructions ### Step 1: Instrumented Firecrawl Wrapper ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); // Counters (use your metrics library: prom-client, statsd, datadog, etc.) function emit(metric: string, value: number, tags?: Record<string, string>) { console.log(JSON.stringify({ metric, value, tags, timestamp: Date.now() })); } export async function instrumentedScrape(url: string) { const start = Date.now(); try { const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"], onlyMainContent: true, }); const duration = Date.now() - start; const quality = ev...