snowflake-incident-runbook

# Snowflake Incident Runbook ## Overview Rapid incident response procedures for Snowflake infrastructure, pipeline failures, and query issues. ## Severity Levels | Level | Definition | Response Time | Examples | |-------|------------|---------------|----------| | P1 | Complete outage | < 15 min | All queries failing, auth broken | | P2 | Degraded service | < 1 hour | High latency, task failures | | P3 | Minor impact | < 4 hours | Snowpipe delays, non-critical errors | | P4 | No user impact | Next business day | Monitoring gaps, cost anomalies | ## Quick Triage (First 5 Minutes) ### Step 1: Is Snowflake Itself Down? ```bash # Check Snowflake status page curl -s https://status.snowflake.com/api/v2/summary.json | python3 -c " import sys, json data = json.load(sys.stdin) print(f\"Status: {data['status']['description']}\") for c in data['components']: if c['status'] != 'operational': print(f\" DEGRADED: {c['name']} - {c['status']}\") " ``` ### Step 2: Can We Connect? ```sql -- Quick connectivity test SELECT CURRENT_TIMESTAMP(), CURRENT_ACCOUNT(), CURRENT_REGION(); -- If this fails, the issue is connectivity/auth, not query logic ``` ### Step 3: What's Failing? ```sql -- Recent failures (last 30 minutes) SELECT error_code, error_message, COUNT(*) AS occurrences, MIN(start_time) AS first_seen, MAX(start_time) AS last_seen FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY WHERE execution_status = 'FAIL' AND start_time >= DATEADD(minutes, -30, CURRENT_TIMEST...