databricks-cost-tuning

# Databricks Cost Tuning ## Overview Reduce Databricks spending through cluster policies, spot instances, SQL warehouse right-sizing, and cost governance. Databricks charges per DBU (Databricks Unit) with rates varying by compute type: Jobs Compute (~$0.15/DBU), All-Purpose Compute (~$0.40/DBU), SQL Compute (~$0.22/DBU), Serverless (~$0.07/DBU). System tables (`system.billing.usage` and `system.billing.list_prices`) provide cost visibility. ## Prerequisites - Databricks Premium or Enterprise workspace - Access to `system.billing.usage` and `system.billing.list_prices` tables - Workspace admin for cluster policy creation ## Instructions ### Step 1: Identify Top Cost Drivers ```sql -- Top 10 most expensive resources this month SELECT cluster_id, COALESCE(usage_metadata.cluster_name, 'unnamed') AS cluster_name, sku_name, SUM(usage_quantity) AS total_dbus, ROUND(SUM(usage_quantity * p.pricing.default), 2) AS estimated_cost_usd FROM system.billing.usage u LEFT JOIN system.billing.list_prices p ON u.sku_name = p.sku_name WHERE u.usage_date >= date_trunc('month', current_date()) GROUP BY cluster_id, cluster_name, u.sku_name ORDER BY estimated_cost_usd DESC LIMIT 10; -- Cost by team (requires cluster tags) SELECT usage_metadata.cluster_tags.Team AS team, sku_name, ROUND(SUM(usage_quantity), 1) AS total_dbus, ROUND(SUM(usage_quantity * p.pricing.default), 2) AS cost_usd FROM system.billing.usage u LEFT JOIN system.billing.list_pric...