data-lake-platform

Featured

Designs lakehouse platforms across Iceberg, Delta, Hudi, and Paimon. Use when choosing catalogs, CDC paths, query engines, governance, or cost controls.

AI & Automation 80 stars 17 forks Updated 1 weeks ago MIT

Install

View on GitHub

Quality Score: 89/100

Stars 20%
64
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Data Lake Platform Build and operate production data lakes and lakehouses: ingest, transform, store in open formats, and serve analytics reliably. ## Quick Reference | Task | Resource | When to Use | |------|----------|-------------| | Pick a table format and spec version | [references/storage-formats.md](references/storage-formats.md) | Choosing Iceberg v2/v3, Delta 4.x, Hudi, Paimon, or DuckLake | | Pick a catalog / control plane | [references/governance-catalog.md](references/governance-catalog.md) | Choosing Polaris, Glue, Nessie, Unity, Gravitino, or Open Catalog | | Design ingestion or CDC path | [references/ingestion-patterns.md](references/ingestion-patterns.md) | dlt, Airbyte, Debezium, Flink CDC | | Scaffold or inspect an Iceberg table | `scripts/scaffold_iceberg_table.py`, `scripts/inspect_iceberg_metadata.sh` | New table DDL or auditing an existing table's file/metadata layout | | Sanity-check an interoperability or version claim | `data/sources.json` + [Fact-Checking](#fact-checking) below | Any claim about spec-version GA status, engine support, or vendor feature | ### Quick-Start Decision Table | Situation | Default choice | |-----------|----------------| | Open multi-engine analytics | Iceberg + REST catalog (Polaris/Glue/Nessie) + Trino | | Databricks primary compute | Delta 4.x + Unity Catalog; add UniForm only if external readers exist | | CDC-heavy mutable, Spark-centered | Hudi (CoW or MoR) + Kafka/Debezium | | Streaming-first mutable, Flink-center...

Details

Author
vasilyu1983
Repository
vasilyu1983/AI-Agents-public
Created
9 months ago
Last Updated
1 weeks ago
Language
Python
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

data-engineering-guidelines

Use when designing, building, or reviewing data-engineering work — lakehouse table design (Delta/Iceberg, medallion, partitioning, compaction, schema evolution, MERGE/CDC), realtime and streaming pipelines (Kafka, Spark Structured Streaming, Flink, watermarks, exactly-once), data governance, AI/ML data support (feature stores, vector/RAG), and data quality, reliability, and cost.

1 Updated 3 days ago
stevencarpenter
AI & Automation Featured

data-streaming

Designs streaming platforms for Kafka, Flink, CDC, and lakehouse ingestion. Use when planning event backbones, CDC pipelines, schema governance, or real-time lakehouse delivery.

80 Updated 1 weeks ago
vasilyu1983
Data & Documents Listed

health-data-lake

When the user wants to design, build, or operate a clinical / healthcare data lake, lakehouse, or warehouse. Use when the user mentions "health data lake," "clinical data warehouse," "healthcare lakehouse," "OMOP," "PCORnet CDM," "Sentinel CDM," "i2b2," "CMS BCDA," "OHDSI," "ATLAS," "HADES," "Athena vocabulary," "FHIR Bulk Data," "$export," "flat FHIR," "SQL-on-FHIR," "Pathling," "Epic Clarity," "Caboodle," "Cerner Millennium ETL," "EMPI," "data quality dashboard," "Achilles," "tokenization vault," "Delta Lake," "Iceberg," "bronze/silver/gold," "Unity Catalog," "Lake Formation," "Snowflake healthcare," "Databricks Lakehouse for Healthcare," or "HIPAA-eligible warehouse." For analytics on top of curated data, see population-health-analytics or clinical-research. For raw FHIR API integration, see fhir-integration. For HL7 v2 ingestion specifics, see hl7-v2.

1 Updated 1 weeks ago
aks-builds