running-chaos-tests

# Chaos Engineering Toolkit ## Overview Execute controlled chaos engineering experiments to test system resilience, fault tolerance, and recovery capabilities. Injects failures including network latency, service crashes, resource exhaustion, and dependency outages to verify that systems degrade gracefully and recover automatically. ## Prerequisites - Distributed system or microservice architecture deployed in a staging/test environment - Monitoring and alerting configured (Grafana, Datadog, CloudWatch, or Prometheus) - Rollback capability for the target environment (manual or automated) - Chaos engineering tool installed (toxiproxy, Pumba, Litmus, or Chaos Mesh) - Explicit approval from the team to run chaos experiments - Steady-state hypothesis defined (what "healthy" looks like in metrics) ## Instructions 1. Define the steady-state hypothesis: - Identify measurable indicators of normal system behavior (e.g., p99 latency < 500ms, error rate < 0.1%, all health checks pass). - Record baseline metrics before injecting any failures. - Define the blast radius -- which services and users are affected by the experiment. 2. Design chaos experiments by category: - **Network**: Inject latency (200-2000ms), packet loss (5-50%), DNS failure, connection timeout. - **Process**: Kill a service instance, exhaust CPU or memory, fill disk. - **Dependency**: Block access to database, cache, or external API. - **State**: Corrupt data, introduce clock skew, simulate sp...