firecrawl-reference-architecture

Featured

Implement Firecrawl reference architecture with scrape/crawl/map/extract pipelines. Use when designing new Firecrawl integrations, reviewing project structure, or building content ingestion pipelines for AI/RAG applications. Trigger with phrases like "firecrawl architecture", "firecrawl project structure", "firecrawl pipeline", "firecrawl RAG", "firecrawl knowledge base".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Firecrawl Reference Architecture ## Overview Production architecture for web scraping and content ingestion with Firecrawl. Covers three tiers: on-demand scraping, scheduled crawl pipelines, and real-time RAG ingestion. Uses all four Firecrawl endpoints: scrape, crawl, map, and extract. ## Architecture Diagram ``` ┌─────────────────────────────────────────────────────────┐ │ Firecrawl Pipeline │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────┐ ┌───────────┐ │ │ │ scrapeUrl│ │ crawlUrl │ │mapUrl│ │ extract │ │ │ │ (1 page) │ │ (N pages)│ │(URLs)│ │ (LLM+JSON)│ │ │ └────┬─────┘ └────┬─────┘ └──┬───┘ └─────┬─────┘ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ Content Processing Layer │ │ │ │ Clean MD │ Validate │ Deduplicate │ Chunk │ │ │ └─────────────────────┬─────────────────────────────┘ │ │ │ │ │ ┌─────────────────────┴─────────────────────────────┐ │ │ │ Storage & Output │ │ │ │ Files │ Database │ Vector Store │ Search Index │ │ │ └───────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` ## Instructions ##...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category