observability-testinglisted

# Observability & Production Testing ## The core skill: correlate, don't guess When something fails, follow the chain: **test/user report → logs → distributed trace → the specific API call → database query → infra metric**. Each link either confirms or rules out a layer. Don't speculate about root cause without walking this chain — "probably a timeout" is not a finding, a trace showing a 30s span on a specific downstream call is. ## What to check per signal type - **Logs**: structured (JSON, not free-text — free-text logs are unqueryable at scale), correlation/request ID present and threaded through every service a request touches, log level appropriate (an error path logging at `info` gets missed; expected/handled conditions logging at `error` cause alert fatigue), no PII/secrets logged. - **Metrics**: the four golden signals present per service — latency, traffic, errors, saturation. Check that a metric actually changes under real load (a counter that's always zero because it's wired to the wrong code path is a silent observability gap, not a healthy system). - **Traces**: spans cover the actual critical path (DB calls, external API calls, queue publish/consume) not just the top-level request; trace context propagates across service/process boundaries (a broken propagation shows up as disconnected trace fragments instead of one end-to-end trace — this is one of the most common and most damaging observability bugs). ## Verifying a feature is observable (do this at feature