A test review compared 20 metrics between two versions of a page. At alpha = 0.05, even if nothing changed, about one metric in twenty will look significant by chance. The more you test, the more false winners you find.
Adjustments keep this in check:
- Bonferroni multiplies each p-value by the number of tests (or divides alpha by it). Simple and strict.
- Holm steps through the sorted p-values with a slightly looser rule. It never rejects less than Bonferroni.
- Benjamini-Hochberg (FDR) controls the share of false discoveries among the winners, a better fit when you screen many metrics.
statsmodels: multipletests(p, alpha=0.05, method='bonferroni') (or 'holm', 'fdr_bh') returns which tests to reject and the adjusted p-values.
Name one primary metric before the test starts; adjust the rest.