incidentlisted

# Incident An incident is a time-compressed debugging problem with real users impacted. The goals in order: restore service, limit blast radius, understand what happened, prevent recurrence. Don't skip steps, but don't let process slow down mitigation either. Move fast on what you know; be explicit about what you don't. ## Phase 1: Triage (first 5 minutes) Establish the facts before taking action: **What is broken?** - Is it a total outage (nothing works) or degradation (some things work, some don't)? - Which services, endpoints, or features are affected? - Are errors specific to a region, user segment, or data condition? **What is the blast radius?** - How many users are impacted? All users, a percentage, a specific cohort? - What is the business impact? (Revenue, SLA breach, data loss risk?) - Is it getting worse, stable, or recovering on its own? **When did it start?** - Look at your error rate and latency graphs and find the inflection point - What changed around that time? Deploys, config changes, traffic patterns, cron jobs, dependency updates? **Set an initial severity.** Be explicit — it determines communication expectations and who needs to be involved: - **SEV1**: customer-facing, widespread impact, revenue or data at risk — all hands - **SEV2**: significant degradation, partial impact — engineering and product engaged - **SEV3**: minor degradation, workaround exists — engineering only ## Phase 2: Communicate early Notify before you've fixed it. Stakeh