databricks-upgrade-migration

# Databricks Upgrade & Migration ## Overview Upgrade Databricks Runtime versions and migrate from Hive Metastore to Unity Catalog. Covers version compatibility, deprecated config removal, table migration via SYNC/CTAS, API endpoint updates, and Delta protocol upgrades. ## Prerequisites - Admin access to workspace - Test environment (dev/staging) for validation before prod - Inventory of current workloads and dependencies ## Instructions ### Step 1: Runtime Version Upgrade #### Version Compatibility Matrix | Current DBR | Target DBR | Key Changes | Effort | |-------------|------------|-------------|--------| | 12.x LTS | 13.3 LTS | Spark 3.4, Python 3.10 default | Low | | 13.3 LTS | 14.3 LTS | Spark 3.5, improved AQE, Liquid Clustering GA | Medium | | 14.x | 15.x LTS | Unity Catalog mandatory, legacy DBFS deprecated | High | #### Automated Upgrade Script ```python from databricks.sdk import WorkspaceClient w = WorkspaceClient() def plan_cluster_upgrade( cluster_id: str, target_version: str = "14.3.x-scala2.12", dry_run: bool = True, ) -> dict: """Plan and optionally execute a DBR version upgrade.""" cluster = w.clusters.get(cluster_id) plan = { "cluster_id": cluster_id, "cluster_name": cluster.cluster_name, "current_version": cluster.spark_version, "target_version": target_version, "removals": [], "warnings": [], } # Check for deprecated Spark configs deprecated = { "spark.data...