vastai-incident-runbook

# Vast.ai Incident Runbook ## Overview Rapid incident response procedures for Vast.ai GPU instance failures. Covers triage, mitigation, recovery, and postmortem for common incident types: spot preemption, instance crashes, GPU failures, and billing issues. ## Prerequisites - Vast.ai CLI access - SSH access to instances (if still running) - Checkpoint storage accessible (S3/GCS) ## Instructions ### Triage: Assess Impact (< 2 minutes) ```bash #!/bin/bash set -euo pipefail echo "=== INCIDENT TRIAGE ===" echo "Time: $(date -u)" # 1. Check all instances echo -e "\n--- Instance Status ---" vastai show instances --raw | python3 -c " import sys, json for inst in json.load(sys.stdin): status = inst.get('actual_status', '?') flag = 'ALERT' if status in ('error', 'exited', 'offline') else 'OK' print(f' [{flag}] ID:{inst[\"id\"]} Status:{status} ' f'GPU:{inst.get(\"gpu_name\",\"?\")} \${inst.get(\"dph_total\",0):.3f}/hr') " # 2. Check if affected instance has recent logs echo -e "\n--- Recent Logs (last 20 lines) ---" vastai logs ${INSTANCE_ID:-0} --tail 20 2>/dev/null || echo "No logs available" # 3. Check account balance echo -e "\n--- Account ---" vastai show user --raw | python3 -c "import sys,json; u=json.load(sys.stdin); print(f'Balance: \${u.get(\"balance\",0):.2f}')" ``` ### Incident Type 1: Spot Preemption **Symptoms**: Instance status changes from `running` to `exited` or `offline` without user action. ```bash # 1. Verify preemption (not user e...