← ClaudeAtlas

data-lake-platformlisted

Padrões de data lake e lakehouse — ingestão, CDC, Iceberg/Delta/Hudi, Trino/DuckDB, orquestração e governança, self-hosted ou em nuvem. Use ao desenhar ou operar uma plataforma de dados.
joaoguirunas/team-os · ★ 6 · AI & Automation · score 74
Install: claude install-skill joaoguirunas/team-os
# Data Lake Platform Build and operate production data lakes and lakehouses: ingest, transform, store in open formats, and serve analytics reliably. ## When to Use - Design data lake/lakehouse architecture - Set up ingestion pipelines (batch, incremental, CDC) - Build SQL transformation layers (SQLMesh, dbt) - Choose table formats and catalogs (Iceberg, Delta, Hudi) - Deploy query/serving engines (Trino, ClickHouse, DuckDB) - Implement streaming pipelines (Kafka, Flink) - Set up orchestration (Dagster, Airflow, Prefect) - Add governance, lineage, data quality, and cost controls ## Triage Questions 1. Batch, streaming, or hybrid? What is the freshness SLO? 2. Append-only vs upserts/deletes (CDC)? Is time travel required? 3. Primary query pattern: BI dashboards (high concurrency), ad-hoc joins, embedded analytics? 4. PII/compliance: row/column-level access, retention, audit logging? 5. Platform constraints: self-hosted vs cloud, preferred engines, team strengths? ## Default Baseline (Good Starting Point) - Storage: object storage + open table format (usually Iceberg) - Catalog: REST/Hive/Glue/Nessie/Unity (match your platform) - Transforms: SQLMesh or dbt (pick one and standardize) - Lake query: Trino (or Spark for heavy compute/ML workloads) - Serving (optional): ClickHouse/StarRocks/Doris for low-latency BI - Governance: DataHub/OpenMetadata + OpenLineage - Orchestration: Dagster/Airflow/Prefect ## Workflow 1. Pick table format + catalog: `references/storage-formats.