chaos-engineering

# Skill: Chaos Engineering > **Expertise:** LitmusChaos experiments, manual K8s chaos, network partition testing, graceful degradation validation. ## When to load When designing chaos experiments, validating failover behavior, verifying SLO headroom, or onboarding a service to chaos testing. ## Chaos Experiment Design Principles ``` 1. Define steady state first → What does "working" look like? (SLI baseline: error rate < 0.1%, p99 < 200ms) 2. Hypothesize → "If 1/3 of pods die, the service will continue serving with p99 < 500ms" 3. Blast radius control → Start with staging. Start with 1 pod. Increase gradually. 4. Abort conditions → Auto-stop if error rate > 1% or p99 > 1s for > 2 min 5. Document and act → Passed = evidence of resilience. Failed = fix + re-test. Never just accept failure. ``` ## Manual Chaos (no tooling needed) ```bash # ── Pod kill (test restart recovery) ────────────────────────── kubectl delete pod <pod-name> -n production # Watch: kubectl get pods -n production -l app=my-service -w # Expected: new pod starts, readiness probe passes, 0 user-visible errors # ── Kill all pods in deployment (test rolling restart recovery) ── kubectl rollout restart deployment/my-service -n production # Watch error rate during rollout # ── Simulate OOMKill ────────────────────────────────────────── kubectl exec -it <pod> -n production -- sh -c \ "dd if=/dev/zero of=/dev/shm/blob bs=1M count=600" # Expected: pod OOMKilled, restarted, alert fired, no...