incrementality-testinglisted

# Incrementality Testing ## Overview Platform ROAS is correlational. The platform optimises toward people who were going to convert anyway and then bills you for the credit. Incrementality is the causal question, and answering it takes three things almost nobody does: 1. **Check the randomisation before reading the result.** 2. **Use a test that stays valid when you peek**, because you will peek. 3. **Reduce variance with data you already have**, instead of buying more users. ## When to use - Designing an experiment on a creative, landing page, channel or product change - Choosing a sample size, or explaining why a test cannot answer the question - Reading a result that has been watched daily - Deciding whether "not significant" means no effect or not enough data - Measuring the incrementality of a channel where you cannot randomise users - Converting platform-reported conversions into incremental ones **Not for:** attribution modelling, which allocates credit among touchpoints that already happened. Attribution splits a pie; incrementality asks how big the pie would have been without you. ## The order is not optional ```bash # 1. is the split even valid? python3 scripts/lift.py srm --arms "control=50120,treatment=49880" # 2. what is the effect? python3 scripts/lift.py lift --control 50120/1204 --treatment 49880/1330 # 3. ...but only if you committed to that sample size. If you have been watching: python3 scripts/lift.py msprt --control 50120/1204 --treatment 49880/