groq-incident-runbook

# Groq Incident Runbook ## Overview Rapid incident response procedures for Groq API failures. Groq is a third-party inference provider -- when it goes down, your mitigation options are: wait, fall back to a different model, or fall back to a different provider. ## Severity Levels | Level | Definition | Response Time | Examples | |-------|------------|---------------|----------| | P1 | Complete API failure | < 15 min | Groq API returns 5xx on all models | | P2 | Degraded performance | < 1 hour | High latency, partial 429s, one model down | | P3 | Minor impact | < 4 hours | Intermittent errors, non-critical feature affected | | P4 | No user impact | Next business day | Monitoring gap, cost anomaly | ## Quick Triage (Run First) ```bash set -euo pipefail echo "=== 1. Groq API Status ===" curl -sf https://status.groq.com > /dev/null && echo "status.groq.com: REACHABLE" || echo "status.groq.com: UNREACHABLE" echo "" echo "=== 2. API Authentication ===" HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \ https://api.groq.com/openai/v1/models \ -H "Authorization: Bearer $GROQ_API_KEY") echo "GET /models: HTTP $HTTP_CODE" echo "" echo "=== 3. Model Availability ===" for model in "llama-3.1-8b-instant" "llama-3.3-70b-versatile"; do CODE=$(curl -s -o /dev/null -w "%{http_code}" \ https://api.groq.com/openai/v1/chat/completions \ -H "Authorization: Bearer $GROQ_API_KEY" \ -H "Content-Type: application/json" \ -d "{\"model\":\"$model\",\"messages\":[{\"role\"...