measure-experiment-design

Featured

Designs an A/B test or experiment with variants, success metrics, sample size, and duration for an existing hypothesis. Use when planning an experiment to validate a product change or test an assumption you have already framed. To articulate the hypothesis itself first, use define-hypothesis.

Web & Frontend 557 stars 73 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 94/100

Stars 20%
91
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

<!-- PM-Skills | https://github.com/product-on-purpose/pm-skills | Apache 2.0 --> # Experiment Design An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal. ## When to Use - Before launching an A/B test to validate a product change - When testing a hypothesis that requires quantitative validation - After solution design to validate assumptions before full rollout - When stakeholders want data-driven evidence for a decision - To establish a culture of experimentation and learning ## When NOT to Use - The hypothesis itself is not yet articulated -> use `define-hypothesis` first; this skill designs the test for a claim you already have - You are analyzing a completed experiment -> use `measure-experiment-results` - You need the event tracking that will measure the experiment -> use `measure-instrumentation-spec` - You are gathering opinions rather than running a controlled test -> use `measure-survey-analysis` ## Instructions When asked to design an experiment, follow these steps: 1. **Articulate the Hypothesis** Write a clear, testable hypothesis in the format: "We believe [change] for [users] will [outcome] as measured by [metric]." O...

Details

Author
product-on-purpose
Repository
product-on-purpose/pm-skills
Created
7 months ago
Last Updated
today
Language
JavaScript
License
Apache-2.0

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category

Web & Frontend Listed

experiment-design

Design hypothesis-driven experiments and A/B tests with proper methodology. Use when asked to design an A/B test, validate a hypothesis, plan an experiment, or set up a test for a product change. Covers hypothesis writing, sample size, and common mistakes.

2 Updated yesterday
AashutoshR2062
AI & Automation Listed

experiment-design

Use when planning A/B tests, fake-door tests, beta rollouts, experiments, metric rules, or interpreting outcomes.

4 Updated 2 weeks ago
Leo-maomao
AI & Automation Featured

experiment-design

A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, running discipline, matching a result to a pre-committed decision rule, and the common failure modes that produce confidently wrong shipping decisions. Use this skill whenever the user is planning a test that has not run yet: framing a hypothesis, sizing the sample, setting duration, choosing guardrails, or deciding whether something is worth testing at all. Triggers on design an experiment, experiment plan, A/B test, split test, multivariate test, holdout, experiment hypothesis, sample size, minimum detectable effect, MDE, test duration, guardrail metric, no peeking, pre-committed decision rule, is this worth testing. Use `experimentation-analytics` instead when the test has already run and the question is how to read the result panel.

567 Updated 1 weeks ago
rampstackco