databricks-migration-deep-dive

# Databricks Migration Deep Dive ## Overview Comprehensive migration strategies for moving to Databricks from Hadoop, Snowflake, Redshift, Synapse, or legacy data warehouses. Covers discovery and assessment, schema conversion, data migration with batching and validation, ETL/pipeline conversion, and cutover planning with rollback procedures. ## Prerequisites - Access to source and target systems - Databricks workspace with Unity Catalog enabled - Understanding of current data architecture and dependencies - Stakeholder alignment on migration timeline ## Migration Patterns | Source | Pattern | Complexity | Timeline | |--------|---------|------------|----------| | Hive Metastore (same workspace) | SYNC / CTAS / DEEP CLONE | Low | Days | | On-prem Hadoop/HDFS | Lift-and-shift to cloud storage + UC | High | 6-12 months | | Snowflake | Parallel run + cutover | Medium | 3-6 months | | AWS Redshift | Unload to S3 + Auto Loader | Medium | 3-6 months | | Legacy DW (Oracle/Teradata) | Full rebuild with JDBC extraction | High | 12-18 months | ## Instructions ### Step 1: Discovery and Assessment Inventory all source tables with metadata for migration planning. ```python from pyspark.sql import SparkSession from dataclasses import dataclass spark = SparkSession.builder.getOrCreate() @dataclass class TableInventory: database: str table: str table_type: str format: str row_count: int size_mb: float columns: int partitions: list[str] def assess_hive_...