firecrawl-policy-guardrails

# Firecrawl Policy Guardrails ## Overview Automated guardrails for Firecrawl scraping pipelines. Web scraping carries legal (robots.txt, ToS), ethical (rate limiting, attribution), and cost (credit burn) risks. This skill implements domain blocklists, credit budgets, content quality gates, and per-domain rate limits as enforceable policies. ## Instructions ### Step 1: Domain Policy Enforcement ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); class ScrapePolicy { // Domains that explicitly prohibit scraping in their ToS static BLOCKED_DOMAINS = [ "facebook.com", "instagram.com", // Meta ToS "linkedin.com", // LinkedIn ToS "twitter.com", "x.com", // X/Twitter ToS ]; // Domains with sensitive/regulated content static SENSITIVE_DOMAINS = [ "*.gov", "*.mil", // Government "*.edu", // Educational (FERPA) ]; static validateUrl(url: string): void { const hostname = new URL(url).hostname; for (const blocked of this.BLOCKED_DOMAINS) { if (hostname === blocked || hostname.endsWith(`.${blocked}`)) { throw new PolicyViolation(`Domain "${hostname}" is blocked: ToS prohibits scraping`); } } for (const pattern of this.SENSITIVE_DOMAINS) { const regex = new RegExp("^" + pattern.replace("*.", ".*\\.") + "$"); if (regex.test(hostname)) { ...