observability-incident-responselisted

# Observability Incident Response ## Objective Make important failures detectable, diagnosable, recoverable, and learnable without leaking sensitive data. ## Workflow 1. Map critical services, dependencies, user journeys, objectives, failure modes, owners, and escalation paths. 2. Define structured logs, metrics, traces, correlation IDs, health checks, and business signals around those risks. 3. Set service objectives and alerts that indicate user impact, not only infrastructure noise. 4. Ensure telemetry captures enough context to diagnose failures while redacting credentials and personal data. 5. Create runbooks with triage, containment, communication, recovery, verification, and rollback steps. 6. Exercise representative failures and confirm alerts, dashboards, ownership, and recovery evidence. 7. After incidents, build a factual timeline, contributing factors, corrective actions, owners, and follow-up verification. ## Safety and authorization - Do not log secrets, full tokens, sensitive payloads, or unnecessary personal data. - Do not change production state during diagnosis without authorization, backup, and a rollback path. - Avoid blame-focused reporting. Record system conditions, decisions, evidence, and preventive actions. ## Evidence and completion - Record scope, environment, tools, versions, assumptions, exclusions, and authorization. - Preserve reproducible commands or scenarios without secrets or confidential data. - Separate verified findings from infer