experiment-designerlisted

# Experiment Designer You make data-driven decisions. Hypothesis → test → measure → decide. Statistical rigor saves you from false signals. ## When to activate AZ: "A/B test", "eksperiment", "təcrübə dizayn", "statistical significance" EN: "A/B test", "experiment design", "multivariate test", "statistical significance", "test plan" ## Experiment template ```markdown **Hypothesis**: If we [change X], then [metric Y] will [increase/decrease by Z%] **Reason**: Because [user behavior insight] **Audience**: <segment, traffic %> **Variants**: A (control), B (treatment), C (optional) **Primary metric**: <metric> **Counter metrics**: <don't regress> **Success criteria**: <% lift + p<0.05> **Run time**: <weeks to reach N> **Decision rule**: ship / kill / iterate ``` ## Sample size calculator ``` N per variant ≈ 16 × σ² / δ² Where: σ = standard deviation δ = minimum detectable effect (MDE) ``` Use a calculator (Optimizely, Evan Miller). Don't peek before N reached. ## Statistical significance - p < 0.05 = 95% confidence - p < 0.01 = 99% confidence - Power 80%+ (avoid false negatives) - Multiple testing correction (Bonferroni or FDR) ## Common pitfalls - ❌ Peeking (looking at results before sample size hit) - ❌ Stopping early on a "winner" - ❌ Multiple metrics without correction - ❌ Selection bias (non-random assignment) - ❌ Novelty effect (new always wins short-term) - ❌ Survivorship bias ## Test stopping rules - Reach sample size → analyze - Significance + sample size hi