databricks-observability

# Databricks Observability ## Overview Monitor Databricks jobs, clusters, SQL warehouses, and costs using system tables in the `system` catalog. System tables provide queryable observability data: `system.lakeflow` (job runs), `system.billing` (costs), `system.query` (SQL history), `system.access` (audit logs), and `system.compute` (cluster metrics). Data updates throughout the day, not real-time. ## Prerequisites - Databricks Premium or Enterprise with Unity Catalog enabled - Access to `system.billing`, `system.lakeflow`, `system.query`, and `system.access` schemas - SQL warehouse for running monitoring queries ## Instructions ### Step 1: Job Health Monitoring ```sql -- Job success/failure over last 24 hours SELECT COUNT(CASE WHEN result_state = 'SUCCESS' THEN 1 END) AS succeeded, COUNT(CASE WHEN result_state = 'FAILED' THEN 1 END) AS failed, COUNT(CASE WHEN result_state = 'TIMED_OUT' THEN 1 END) AS timed_out, ROUND(100.0 * COUNT(CASE WHEN result_state = 'SUCCESS' THEN 1 END) / COUNT(*), 1) AS success_rate_pct, ROUND(AVG(TIMESTAMPDIFF(MINUTE, start_time, end_time)), 1) AS avg_duration_min FROM system.lakeflow.job_run_timeline WHERE start_time > current_timestamp() - INTERVAL 24 HOURS; -- Failed jobs with error details SELECT job_id, run_name, result_state, start_time, end_time, TIMESTAMPDIFF(MINUTE, start_time, end_time) AS duration_min, error_message FROM system.lakeflow.job_run_timeline WHERE result_state = 'FAILED' AND start_time >...