debug-a-production-incidentlisted

# debug-a-production-incident Run an incident as two connected jobs: reduce harm now, then learn enough to prevent a repeat. Prefer safe restoration over a clever diagnosis while users are still affected. Keep one factual timeline so responders do not have to reconstruct the event from chat. ## When to use Use this skill when users are failing, data may be wrong, an important service-level signal is breached, or responders must coordinate a production investigation. Do not declare an incident for a local development bug with no shared or production effect. Never expose customer data, credentials, or private logs in a public channel. Do not make several uncontrolled changes at once; each change needs an owner, timestamp, and observed result. ## Preconditions - Open an incident record with a lead, communications owner, severity, and start time. - Identify the safest source of production signals and the authority required for changes. - Preserve relevant logs, traces, deploy identifiers, feature-flag state, and recent change history. - Agree on a communication interval and the audience for internal and external updates. ## Procedure 1. State the user-visible symptom, affected population, first known time, and current severity. 2. Start an incident timeline. Record observations and actions separately from hypotheses. 3. Check broad dependencies and recent changes before diving into one component. 4. Choose the safest containment: disable a feature, shed optional work, fai