ab-testing

# A/B testing — design and read a defensible experiment An experiment without a pre-committed sample size and a single primary metric is not an experiment. It is a dashboard you stare at until it tells you what you wanted to hear. The discipline lives almost entirely *before* traffic ships: a falsifiable hypothesis, one primary metric, a sample size derived from the smallest effect worth detecting, and a stop rule you cannot renegotiate at 2pm on day four. ## Pre-test checklist — every line true before any traffic Each one is a place experiments die silently. - [ ] A **falsifiable hypothesis** — names the change, the direction, and the metric it moves. - [ ] Exactly **ONE primary metric**. More than one primary = multiple comparisons = inflated false positives. - [ ] **Guardrail metrics** — what you refuse to harm (latency, refunds, unsubscribes) even for a win. - [ ] The **randomization unit = the analysis unit** (usually the user). Mixing them is pseudoreplication. - [ ] An **MDE** — the smallest lift that would change a decision. Not "any difference." - [ ] A **computed sample size** and the **duration** it implies at your real daily eligible traffic. - [ ] A **fixed stop rule** — a date or an n you commit to before launch. No "we'll see how it looks." ## Step 1 — Hypothesis and metrics State a null you can reject. "The new checkout button changes purchase conversion" with H0: conversion equal across arms, H1: it differs. Vague aspirations ("improve the funnel") have...