experiment-designer

# Experiment Designer Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions. ## When To Use Use this skill for: - A/B and multivariate experiment planning - Hypothesis writing and success criteria definition - Sample size and minimum detectable effect planning - Experiment prioritization with ICE scoring - Reading statistical output for product decisions ## Core Workflow 1. Write hypothesis in If/Then/Because format - If we change `[intervention]` - Then `[metric]` will change by `[expected direction/magnitude]` - Because `[behavioral mechanism]` 2. Define metrics before running test - Primary metric: single decision metric - Guardrail metrics: quality/risk protection - Secondary metrics: diagnostics only 3. Estimate sample size - Baseline conversion or baseline mean - Minimum detectable effect (MDE) - Significance level (alpha) and power Use: ```bash python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute ``` 4. Prioritize experiments with ICE - Impact: potential upside - Confidence: evidence quality - Ease: cost/speed/complexity ICE Score = (Impact * Confidence * Ease) / 10 5. Launch with stopping rules - Decide fixed sample size or fixed duration in advance - Avoid repeated peeking without proper method - Monitor guardrails continuously 6. Interpret results - Statistical significance is not business significance - Compare point estimate + confidence interval to decision threshol...