data-lake-platformlisted
Install: claude install-skill joaoguirunas/team-os
# Data Lake Platform
Build and operate production data lakes and lakehouses: ingest, transform, store in open formats, and serve analytics reliably.
## When to Use
- Design data lake/lakehouse architecture
- Set up ingestion pipelines (batch, incremental, CDC)
- Build SQL transformation layers (SQLMesh, dbt)
- Choose table formats and catalogs (Iceberg, Delta, Hudi)
- Deploy query/serving engines (Trino, ClickHouse, DuckDB)
- Implement streaming pipelines (Kafka, Flink)
- Set up orchestration (Dagster, Airflow, Prefect)
- Add governance, lineage, data quality, and cost controls
## Triage Questions
1. Batch, streaming, or hybrid? What is the freshness SLO?
2. Append-only vs upserts/deletes (CDC)? Is time travel required?
3. Primary query pattern: BI dashboards (high concurrency), ad-hoc joins, embedded analytics?
4. PII/compliance: row/column-level access, retention, audit logging?
5. Platform constraints: self-hosted vs cloud, preferred engines, team strengths?
## Default Baseline (Good Starting Point)
- Storage: object storage + open table format (usually Iceberg)
- Catalog: REST/Hive/Glue/Nessie/Unity (match your platform)
- Transforms: SQLMesh or dbt (pick one and standardize)
- Lake query: Trino (or Spark for heavy compute/ML workloads)
- Serving (optional): ClickHouse/StarRocks/Doris for low-latency BI
- Governance: DataHub/OpenMetadata + OpenLineage
- Orchestration: Dagster/Airflow/Prefect
## Workflow
1. Pick table format + catalog: `references/storage-formats.