canary-analysislisted

# Canary analysis A canary sends the new version to a small slice of live traffic and compares its behavior against the old version, so a regression hits a fraction of users instead of all of them. The judgment goes wrong when the comparison is unfair, the sample is too small to mean anything, or a human eyeballs a green dashboard and ships on the first good-looking minute. Rigor here is the difference between catching a regression and blessing one. ## Method 1. **Compare against a concurrent baseline, not history.** Run the old version as a control on the same population in the same window, and diff the canary against it. Comparing to yesterday's numbers lets time-of-day, traffic mix, and weekend effects masquerade as a regression or hide a real one. 2. **Set metric gates before the deploy.** Pick error rate, latency at p50 and p99, CPU and memory, and one business metric such as checkout rate; declare the direction and threshold for each up front. Tools like Spinnaker's Kayenta score these into a single verdict. 3. **Wait for a minimum sample before scoring.** A canary on 1% of traffic for ninety seconds cannot detect a 2% regression. Require a minimum request count or duration per gate, and refuse to render a verdict until it is met. 4. **Stay statistically honest.** Do not peek and ship the instant it looks green: continuously re-checking inflates false positives. Use a fixed window or a sequential test built for repeated looks, and correc