quantitative-analysis

# Quantitative analysis A wrong answer here is rarely an arithmetic error. It is a correct calculation on the wrong comparison, or on data that does not mean what the field name suggests. ## Frame the question so that an answer changes something Start from the decision. "How is retention doing" has no answer; "is the cohort we changed onboarding for retaining better than the one before it, enough to justify rolling it out" does. Write down what you expect to find and what you would do in each case before you look. If every possible result leads to the same action, the analysis is not worth running — and knowing that in advance is worth more than the analysis would have been. ## Choose the comparison before the metric Almost every meaningful number is a comparison, and the choice of what to compare against does more work than the calculation. - **Against what it was** — needs a period long enough to see through seasonality and noise. - **Against what it would have been** — the strongest comparison and the hardest to construct. A holdout group, a matched segment, a pre-trend extended forward. - **Against a peer or a benchmark** — only useful if the definitions genuinely match, which they usually do not. **Name the counterfactual explicitly.** "Revenue rose after the campaign" is a comparison against nothing, and it is the single most common way credit is claimed for a trend that was already happening. ## Interrogate the data before you trust it Look at the raw row...