firecrawl-policy-guardrails

Featured

Implement Firecrawl scraping policy enforcement: domain blocklists, credit budgets, content filtering, and robots.txt compliance guardrails. Use when setting up scraping policies, enforcing crawl limits, or preventing accidental scraping of prohibited domains. Trigger with phrases like "firecrawl policy", "firecrawl guardrails", "firecrawl domain blocklist", "firecrawl scraping rules", "firecrawl compliance".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Firecrawl Policy Guardrails ## Overview Automated guardrails for Firecrawl scraping pipelines. Web scraping carries legal (robots.txt, ToS), ethical (rate limiting, attribution), and cost (credit burn) risks. This skill implements domain blocklists, credit budgets, content quality gates, and per-domain rate limits as enforceable policies. ## Instructions ### Step 1: Domain Policy Enforcement ```typescript import FirecrawlApp from "@mendable/firecrawl-js"; const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, }); class ScrapePolicy { // Domains that explicitly prohibit scraping in their ToS static BLOCKED_DOMAINS = [ "facebook.com", "instagram.com", // Meta ToS "linkedin.com", // LinkedIn ToS "twitter.com", "x.com", // X/Twitter ToS ]; // Domains with sensitive/regulated content static SENSITIVE_DOMAINS = [ "*.gov", "*.mil", // Government "*.edu", // Educational (FERPA) ]; static validateUrl(url: string): void { const hostname = new URL(url).hostname; for (const blocked of this.BLOCKED_DOMAINS) { if (hostname === blocked || hostname.endsWith(`.${blocked}`)) { throw new PolicyViolation(`Domain "${hostname}" is blocked: ToS prohibits scraping`); } } for (const pattern of this.SENSITIVE_DOMAINS) { const regex = new RegExp("^" + pattern.replace("*.", ".*\\.") + "$"); if (regex.test(hostname)) { ...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category