clerk-incident-runbook

# Clerk Incident Runbook ## Overview Procedures for responding to Clerk-related incidents in production. Covers triage, emergency auth bypass, recovery scripts, and post-incident review. ## Prerequisites - Access to Clerk Dashboard (dashboard.clerk.com) - Access to application logs and monitoring - Emergency contact list for on-call team - Rollback procedures documented ## Instructions ### Step 1: Triage — Identify Incident Category | Category | Symptoms | Severity | |----------|----------|----------| | Clerk outage | status.clerk.com shows incident, all auth fails | Critical | | Key compromise | Unauthorized access detected | Critical | | Middleware failure | All routes return 500 | High | | Session issues | Users randomly logged out | Medium | | Webhook backlog | User sync falling behind | Low | Quick diagnostic: ```bash #!/bin/bash # scripts/clerk-triage.sh set -euo pipefail echo "=== Clerk Incident Triage ===" echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)" # 1. Check Clerk status echo -e "\n--- Clerk Status ---" curl -s https://status.clerk.com/api/v2/status.json | python3 -c " import json,sys d=json.load(sys.stdin) print(f\"Status: {d['status']['description']}\")" 2>/dev/null || echo "Cannot reach status API" # 2. Check API connectivity echo -e "\n--- API Connectivity ---" HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer ${CLERK_SECRET_KEY}" \ https://api.clerk.com/v1/users?limit=1 2>/dev/null) echo "API response: HTTP $HTTP_CODE" # 3...