databricks-incident-runbook

# Databricks Incident Runbook ## Overview Rapid incident response for Databricks: triage script, decision tree, immediate actions by error type, communication templates, evidence collection, and postmortem template. Designed for on-call engineers to follow during live incidents. ## Severity Levels | Level | Definition | Response Time | Examples | |-------|------------|---------------|----------| | P1 | Production pipeline down | < 15 min | Critical ETL failed, data not updating | | P2 | Degraded performance | < 1 hour | Slow queries, partial failures, stale data | | P3 | Non-critical issues | < 4 hours | Dev cluster issues, non-critical job delays | | P4 | No user impact | Next business day | Monitoring gaps, cleanup needed | ## Instructions ### Step 1: Quick Triage (Run First) ```bash #!/bin/bash set -euo pipefail echo "=== DATABRICKS TRIAGE $(date -u +%H:%M:%S\ UTC) ===" # 1. Is Databricks itself down? echo "--- Platform Status ---" curl -s https://status.databricks.com/api/v2/status.json | \ jq -r '.status.description // "UNKNOWN"' # 2. Can we reach the workspace? echo "--- Workspace ---" if databricks current-user me --output json 2>/dev/null | jq -r .userName; then echo "API: CONNECTED" else echo "API: UNREACHABLE — check VPN/firewall/token" fi # 3. Recent failures echo "--- Failed Runs (last 1h) ---" databricks runs list --limit 20 --output json 2>/dev/null | \ jq -r '.runs[]? | select(.state.result_state == "FAILED") | "\(.run_id): \(.run_name /...