cohere-incident-runbook

# Cohere Incident Runbook ## Overview Rapid incident response procedures for Cohere API v2 outages. Covers triage, mitigation, communication, and postmortem for Chat, Embed, Rerank, and Classify endpoints. ## Prerequisites - Access to [status.cohere.com](https://status.cohere.com) - kubectl access to production cluster - Prometheus/Grafana access - PagerDuty/Slack communication channels ## Severity Levels | Level | Definition | Response Time | Example | |-------|------------|---------------|---------| | P1 | All Cohere endpoints down | < 15 min | API returning 5xx globally | | P2 | Degraded (rate limits, high latency) | < 1 hour | 429 errors, P95 > 10s | | P3 | Single endpoint affected | < 4 hours | Embed works, Chat fails | | P4 | Non-blocking issue | Next business day | Slow response, minor errors | ## Quick Triage (Run These First) ```bash # 1. Check Cohere service status curl -s https://status.cohere.com/api/v2/status.json | jq '.status.description' # 2. Test each endpoint directly echo "--- Chat ---" curl -s -o /dev/null -w "%{http_code}" \ -X POST https://api.cohere.com/v2/chat \ -H "Authorization: Bearer $CO_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"command-r7b-12-2024","messages":[{"role":"user","content":"ping"}]}' echo -e "\n--- Embed ---" curl -s -o /dev/null -w "%{http_code}" \ -X POST https://api.cohere.com/v2/embed \ -H "Authorization: Bearer $CO_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"embed-v4....