Who asked: Priya Raman, product manager at Lumen Books, an online bookshop.
Why: the new checkout page (B) has run against the old one (A) for two weeks, and Priya has to decide on Friday whether to ship it.
The data: test_summary, sessions and orders for each page, and checkout_orders, a sample of 20 order values per page.
What done looks like: conversion rates and lift, a significance test for conversion and one for order value, revenue per session per page, and a decision: ship B, keep A, or run longer.
An A/B test splits visitors at random between the old page (A) and a new one (B). Because the split is random, a difference that is bigger than chance would produce can be put down to the page. A good analysis checks the main metric, checks what else moved, and turns both into one number the business cares about. Each step adds to the same script.