timeouts-and-retrieslisted

# Timeouts and retries Every network call needs three decisions made on purpose: how long to wait, whether to try again, and when to stop trying at all. Defaults (infinite, always, never) cause most cascade outages. ## Method 1. **Budget top-down.** The user-facing SLO (say 2s) is the budget; each hop gets a slice, and a caller's timeout must exceed the sum of its downstream's timeout-times-retries. Propagate the remaining budget (deadline header) so deep services stop working on requests the edge has already abandoned. 2. **Set both connect and request timeouts.** Connect short (1s: the host is there or it is not), request from the dependency's p99 plus margin, not from hope. An unset client timeout inherits "forever" from most HTTP libraries; audit every client for it. 3. **Retry only what is safe and useful.** Idempotent operations (GET/PUT/DELETE, POSTs with idempotency keys) on connect errors, timeouts, 429 and 5xx. Never on 4xx (except 429), never on business failures, never blind-retry a POST that may have committed. 4. **Cap retries hard and jitter the backoff.** 2-3 attempts total, exponential backoff with full jitter (`sleep(random(0, base * 2^attempt))`). Retry storms are self-DDoS: at 3 layers each retrying 3 times, one slow query receives 27x traffic. Retry at one layer (usually the edge), not every layer. 5. **Add a retry budget.** Site-wide cap (retries <= 10-20% of requests); when exceeded, stop retrying and fail