firecrawl-local-dev-loop

# Firecrawl Local Dev Loop ## Overview Set up a fast development workflow for Firecrawl integrations. Use self-hosted Firecrawl via Docker to avoid burning API credits during development, mock the SDK for unit tests, and run integration tests against the local instance. ## Prerequisites - Node.js 18+ with npm/pnpm - Docker + Docker Compose (for self-hosted Firecrawl) - `@mendable/firecrawl-js` installed ## Instructions ### Step 1: Project Structure ``` my-firecrawl-project/ ├── src/ │ ├── scraper.ts # Firecrawl business logic │ └── config.ts # Environment-aware config ├── tests/ │ ├── scraper.test.ts # Unit tests (mocked SDK) │ └── integration.test.ts # Integration tests (real API) ├── docker-compose.yml # Self-hosted Firecrawl ├── .env.local # Dev secrets (git-ignored) ├── .env.example # Template for team └── package.json ``` ### Step 2: Self-Hosted Firecrawl for Zero-Credit Dev ```yaml # docker-compose.yml services: firecrawl: image: mendableai/firecrawl:latest ports: - "3002:3002" environment: - PORT=3002 - USE_DB_AUTHENTICATION=false - REDIS_URL=redis://redis:6379 - NUM_WORKERS_PER_QUEUE=1 - BULL_AUTH_KEY=devonly depends_on: redis: condition: service_healthy redis: image: redis:7-alpine ports: - "6379:6379" healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 5s timeout: 3s retries: 5 ``` ``...