data-engineering-guidelineslisted

# Data Engineering Guidelines Shared rubric for data-engineering agents working on lakehouse, realtime/streaming, and AI/ML data systems. Vendor-general — applies to Delta Lake or Apache Iceberg, Databricks or open-source Spark/Flink/Kafka. Understand the data contract and the access pattern before you write a pipeline. When you recommend a single-vendor component (catalog, feature store, orchestration), name at least one viable alternative and say why the pick wins — existing infra, team skillset, cost — never default silently to one vendor's stack. ## Source Of Truth - The **data contract**: schema, semantics, freshness/latency SLA, expected volume and growth, ownership, and downstream consumers. If it isn't written down, write it down first. - The live catalog and lineage (Unity Catalog, Glue, Hive Metastore, or an Iceberg catalog): what exists, who reads it, and how it's governed. - The platform's own behavior — read the engine docs for the exact version. Streaming and table-format semantics change between releases. ## Lakehouse Table Design - **Table format:** Delta Lake or Apache Iceberg, not bare Parquet — you need ACID, schema evolution, and time travel. In 2026 the formats increasingly interoperate (Delta UniForm exposes Iceberg metadata; engines read both through the **Iceberg REST catalog** spec), so decide the **catalog / REST-catalog** story first — the on-disk format is becoming a swappable detail. Still pick one primary format per platform and be consisten