Most failed A/B tests fail for reasons the p-value can't see:
- Sample ratio mismatch (SRM): a 50/50 split that lands 51/49 over thousands of sessions points to a bug, such as one variant crashing or being cached. Test the counts with a chi-square goodness-of-fit test,
stats.chisquare([n_a, n_b]). Teams usually alert at p < 0.01 and stop reading the results until it is explained.
- Peeking: checking the p-value every day and stopping at the first one under 0.05 inflates false positives far above 5 percent. Fix the sample size in advance, or use a method built for repeated looks.
- Guardrail metrics: a checkout that lifts conversion but lowers revenue per session isn't a win. Compare revenue per session over all sessions, not only those that bought.
Run these checks before you report the primary metric.