experiment-designlisted

Design experiments that actually prove something. Most A/B tests fail because they test vague ideas, run too short, or peek at results. A well-designed experiment has a clear hypothesis, adequate power, and a pre-committed analysis plan. ## Hypothesis Template Every experiment starts with a written hypothesis before any work begins: **"If we [make this specific change] for [this audience], then [this metric] will [change in this direction] by [this amount], because [this reason based on evidence]."** Example: > "If we replace the 5-step onboarding wizard with a single guided first-project flow for new signups, then 7-day activation rate will increase from 23% to 35%, because 4/6 interviewed users said they wanted to 'just start using it' not 'set everything up first.'" Every part matters: - **Specific change:** Not "improve onboarding" — the exact change - **Audience:** Who sees this? New users only? Free tier only? - **Metric + direction + amount:** A number you'll measure - **Because:** The evidence-based reason. No evidence = no experiment. ## Experiment Design ### 1. Primary Metric One metric the experiment is designed to move. Not three. One. Additional metrics are guardrails. ### 2. Guardrail Metrics Metrics that must NOT degrade. These prevent "winning" by breaking something else. ### 3. Sample Size Calculate BEFORE running. Use a sample size calculator with: - Baseline conversion rate (current number) - Minimum detectable effect (smallest change worth caring