chaos-and-resiliencelisted

Chaos engineering, resilience patterns, failure recovery, and fault tolerance for the {{PROJECT_NAME}}. Covers circuit breaker patterns (HTTP, WS, per-venue), reconnect gate design, graceful shutdown protocol, backpressure strategies (PubSub 1MB/4MB, persistence queues), worker lifecycle management, IPC channel health detection, failure scenario matrix, bounded queue design, and event loop stall recovery. Use when reviewing or writing any code that touches error handling, retry logic, reconnection, circuit breakers, worker management, IPC channels, queue management, backpressure, graceful shutdown, health checks, or any failure-adjacent code path.
Canhada-Labs/ceo-orchestration · ★ 2 · AI & Automation · score 74

Install: claude install-skill Canhada-Labs/ceo-orchestration

# Chaos and Resilience ## Fail-Fast Rule If a system component enters an unrecoverable state, **fail fast and loud**. Never silently swallow errors in resilience-critical paths. Never assume a retry will fix a structural problem. Never disable safety mechanisms (watchdogs, circuit breakers) without a replacement. ## Current Resilience Posture (Audit 2026-03-23: 6.5/10) ### What Works - Circuit breakers: 3 layers (HTTP, WS, per-venue) - Reconnect gate: max 3 connections per 10s with jitter backoff - Graceful shutdown: SIGINT/SIGTERM with 10s hard kill - Adapter process auto-restart: 10x with exponential backoff - PubSub backpressure: 1MB skip, 4MB force-close, slow-client detection - EL monitoring: stall detector, iteration tracker, per-operation profiler ### What Does NOT Work | Issue | Impact | File | |-------|--------|------| | IPC batch buffers UNBOUNDED | OOM then restart cascade then permanent death | adapter-process.ts:330-351 | | BP watchdog DISABLED | Worker can hang indefinitely | adapter-process.ts:71 | | Max restarts without recovery | After 10 crashes, dies permanently | gateway-wiring.ts | | Supabase without circuit breaker | Flush workers waste CPU when Supabase is down | supabase-persistence.ts | | unshift bypass in backpressure | Queues grow beyond cap | supabase-persistence.ts:877 | | unhandledRejection not fatal | System in inconsistent state continues running | index.ts:2557 | ## Top 5 Failure Scenarios ### 1. IPC Silent Failure (CRITICAL) ``` IPC