data-sourcinglisted

Data collection and readiness — the step BEFORE a pipeline exists. Catalog candidate data sources for an objective, rank each by importance/impact, and record where to get it (internal system vs. open/public dataset), how to access it, its refresh cadence, licensing, and readiness blockers. Use this skill whenever a project is starting and the sources aren't decided yet, when the user asks "what data do I need for X and where do I get it", when weighing internal vs. open-source data, or when assessing whether a source is accessible/usable before committing to a build. This sits upstream of pipeline-design — it decides WHAT to ingest before deciding HOW.
Methasit-Pun/data_engineer_claude_skills · ★ 1 · Data & Documents · score 65

Install: claude install-skill Methasit-Pun/data_engineer_claude_skills

# Data Sourcing & Readiness The step before you build anything. You cannot design a pipeline for data you haven't located, priced, or confirmed you're allowed to use. This skill produces a **source inventory** the rest of the lifecycle consumes. ## Start from the objective, not the data Ask first: *what question or product does this data serve?* Every source is judged by how much it moves that objective — not by how easy it is to grab. 1. **Objective** — BI dashboard? ML model? Regulatory report? One-off analysis? 2. **Decision the data drives** — what changes based on this data? 3. **Minimum viable sources** — the smallest set that answers the objective. Resist collecting "nice to have" data no one will use. ## Build the source inventory For every candidate source, fill one row. This table is the deliverable. | Column | What to capture | |---|---| | Source name | Human name (e.g. "Salesforce Opportunities", "OpenStreetMap") | | Type | Internal system / SaaS API / public dataset / file drop / scrape | | Importance | Critical / important / enrichment — impact on the objective | | Where to get it | Exact system, endpoint, URL, or dataset repository | | Access method | DB credential, API key, OAuth, S3 bucket, download, request-to-owner | | Owner | Team or person who controls access | | Refresh cadence | Real-time / daily / monthly / static | | Volume | Rough row/byte scale — feeds architecture sizing | | Licensing | Internal-only / open license (name it) / restricted / P