experiment-analysis

# Experiment Analysis A rigorous frequentist framework for designing and analyzing A/B tests: state a falsifiable hypothesis, pre-register one primary metric plus guardrails, compute the sample size needed to detect a meaningful effect, then read the result honestly — without peeking or p-hacking — and convert it into a ship/kill decision. **Grounded in:** *Trustworthy Online Controlled Experiments* — Kohavi, Tang & Xu: pre-registered metric, power, and an honest read of significance vs. MDE. **Go deeper (The Product Channel):** [Learning from Experiments](https://sidsaladi.substack.com/p/week-26-learning-from-experiments) ## When to use this - You're about to launch an A/B test and need a design that won't be ambiguous later ("here's the change, help me set it up properly"). - A test has finished (or is mid-flight) and you need to interpret the numbers and decide ship vs. kill. - Someone shows you a "significant" result and you suspect peeking, a tiny sample, or a cherry-picked metric. - You need to estimate how long a test must run, or whether you even have the traffic to detect the effect you care about. - You're refereeing a debate where two people read the same dashboard and reached opposite conclusions. ## Before you start (gather these) - **The change and the mechanism** — what's different in treatment, and *why* you believe it moves the metric. - **Primary metric** — the single conversion/rate/continuous metric that decides the test. One. Plus its current baseline...