graceful-shutdownlisted
Install: claude install-skill Amey-Thakur/AI-SKILLS
# Graceful shutdown
Every deploy kills your processes. Errors that spike at deploy time are
not bad luck; they are an unimplemented shutdown sequence.
## Method
1. **Trap SIGTERM and flip readiness first.** On signal: mark the
readiness/health endpoint unready so the load balancer stops sending
new work, but keep serving in-flight requests. Liveness stays green;
you are draining, not dead.
2. **Wait out the routing lag.** There is a gap between "unready" and
"traffic actually stops" (LB health-check interval, k8s endpoint
propagation). Sleep a few seconds before closing listeners; closing
immediately resets connections the balancer already routed.
3. **Drain in dependency order.** Stop accepting new HTTP connections,
finish in-flight requests, stop pulling new jobs from queues, finish
or checkpoint running jobs, flush buffers (logs, metrics, producer
queues), then close DB pools and connections last; everything above
needs them.
4. **Bound the drain with a deadline.** Grace budget minus routing-lag
sleep is your drain window (k8s `terminationGracePeriodSeconds`
default 30s; set it above your request timeout). At the deadline,
log what was abandoned and exit nonzero context: requests longer than
the grace period must be redesigned as jobs.
5. **Make workers stop between units.** Job loops check a shutdown flag
between items; long items heartbeat and checkpoint so the next worker
resumes instead of restarting (see background-jo