firecrawl-reference-architecture

Featured

Implement Firecrawl reference architecture with scrape/crawl/map/extract pipelines. Use when designing new Firecrawl integrations, reviewing project structure, or building content ingestion pipelines for AI/RAG applications. Trigger with phrases like "firecrawl architecture", "firecrawl project structure", "firecrawl pipeline", "firecrawl RAG", "firecrawl knowledge base".

AI & Automation 2,266 stars 315 forks Updated today MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Firecrawl Reference Architecture ## Overview Production architecture for web scraping and content ingestion with Firecrawl. Covers three tiers: on-demand scraping, scheduled crawl pipelines, and real-time RAG ingestion. Uses all four Firecrawl endpoints: scrape, crawl, map, and extract. ## Architecture Diagram ``` ┌─────────────────────────────────────────────────────────┐ │ Firecrawl Pipeline │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────┐ ┌───────────┐ │ │ │ scrapeUrl│ │ crawlUrl │ │mapUrl│ │ extract │ │ │ │ (1 page) │ │ (N pages)│ │(URLs)│ │ (LLM+JSON)│ │ │ └────┬─────┘ └────┬─────┘ └──┬───┘ └─────┬─────┘ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ Content Processing Layer │ │ │ │ Clean MD │ Validate │ Deduplicate │ Chunk │ │ │ └─────────────────────┬─────────────────────────────┘ │ │ │ │ │ ┌─────────────────────┴─────────────────────────────┐ │ │ │ Storage & Output │ │ │ │ Files │ Database │ Vector Store │ Search Index │ │ │ └───────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` ## Instructions ##...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
7 months ago
Last Updated
today
Language
Python
License
MIT

Integrates with

Anthropic · AI

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Featured

firecrawl-architecture-variants

Choose and implement Firecrawl architecture patterns for different scales and use cases. Use when designing new Firecrawl integrations, choosing between on-demand/scheduled/pipeline architectures, or planning scraping infrastructure. Trigger with phrases like "firecrawl architecture", "firecrawl blueprint", "how to structure firecrawl", "firecrawl at scale", "firecrawl pipeline design".

2,266 Updated today
jeremylongshore
AI & Automation Featured

firecrawl-hello-world

Create a minimal working Firecrawl example that scrapes a page to markdown. Use when starting a new Firecrawl integration, testing your setup, or learning the scrape/crawl/map/extract API surface. Trigger with phrases like "firecrawl hello world", "firecrawl example", "firecrawl quick start", "simple firecrawl code".

2,266 Updated today
jeremylongshore
AI & Automation Listed

firecrawl-automation

Automate web crawling and data extraction with Firecrawl -- scrape pages, crawl sites, extract structured data, batch scrape URLs, and map website structures through the Composio Firecrawl integration.

62,564 Updated 1 weeks ago
ComposioHQ
Data & Documents Listed

firecrawl

Search, scrape, and interact with the web via the Firecrawl CLI. Use this skill whenever the user wants to search the web, find articles, research a topic, look something up online, scrape a webpage, grab content from a URL, get data from a website, crawl documentation, download a site, or interact with pages that need clicks or logins. Also use when they say "fetch this page", "pull the content from", "get the page at https://", or reference external websites. This provides real-time web search with full page content and interact capabilities — beyond what Claude can do natively with built-in tools. Do NOT trigger for local file operations, git commands, deployments, or code editing tasks.

24 Updated yesterday
georgekhananaev
AI & Automation Listed

firecrawl

Scrape pages, crawl public sites, map URLs, and run JSON-schema extraction through managed or self-hosted Firecrawl.

108 Updated today
HybridAIOne