databricks-core-workflow-a

# Databricks Core Workflow A: Delta Lake ETL ## Overview Build production Delta Lake ETL pipelines using the medallion architecture (Bronze > Silver > Gold). Uses Auto Loader (`cloudFiles`) for incremental ingestion, `MERGE INTO` for upserts, and Delta Live Tables for declarative pipelines. ## Prerequisites - Completed `databricks-install-auth` setup - Unity Catalog enabled with catalogs/schemas created - Access to cloud storage for raw data (S3, ADLS, GCS) ## Architecture ``` Raw Sources (S3/ADLS/GCS) │ Auto Loader (cloudFiles) ▼ Bronze (raw + metadata) │ Cleanse, deduplicate, type-cast ▼ Silver (conformed) │ Aggregate, join, feature engineer ▼ Gold (analytics-ready) ``` ## Instructions ### Step 1: Bronze Layer — Raw Ingestion with Auto Loader Auto Loader (`cloudFiles` format) incrementally processes new files as they arrive. It handles schema inference, evolution, and scales to millions of files. ```python from pyspark.sql import SparkSession from pyspark.sql.functions import current_timestamp, input_file_name, lit spark = SparkSession.builder.getOrCreate() # Streaming ingestion with Auto Loader bronze_stream = ( spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", "/checkpoints/bronze/orders/schema") .option("cloudFiles.inferColumnTypes", "true") .option("cloudFiles.schemaEvolutionMode", "addNewColumns") .load("s3://data-lake/raw/orders/") ) # Add ing...