oraclecloud-incident-runbook

# Oracle Cloud Incident Runbook ## Overview Self-service runbook for when OCI instances go down and the status page stays green. OCI's status page has a history of not acknowledging outages in real time (London Jan 2026 — 502s and instances disappearing for 10 minutes with no status update). OCI Support response times average 4+ hours for Sev-1 tickets. This runbook gives you health probes, automated instance recovery, cross-AD failover, and cross-region failover — all executable without waiting on Oracle. **Purpose:** Detect OCI service degradation independently, recover instances automatically, and fail over to alternate availability domains or regions when the primary is impacted. ## Prerequisites - **OCI CLI installed and configured** — `~/.oci/config` validated (see `oraclecloud-install-auth`) - **Python 3.8+** with the OCI SDK — `pip install oci` - **Pre-configured resources**: at least one compute instance, a VCN with subnets in multiple ADs - **IAM policies**: `manage instances`, `manage volumes`, `inspect work-requests` in the target compartment - **Boot volume backups** enabled (recovery depends on having a recent backup) ## Instructions ### Step 1: Independent Health Probes Do not trust the OCI status page alone. Run your own health checks against the OCI API: ```python import oci import time config = oci.config.from_file("~/.oci/config") def probe_oci_health(config): """Probe OCI API endpoints independently of the status page.""" results = {} ...