Case study: an experiment platform
Request
The Maple Cart product owner needs a checkout experiment platform and a rollout decision.
Data
Assignment is by eligible user with a planned 50/50 split. Observed assignments are A 6,000 and B 4,000; B has 3,200 exposures and 240 completed orders. The checkout-error guardrail failed. These are fictional exercise inputs, not research findings.
Done
Deliver stable assignment settings, the assignment model, the observed metrics, and a checked recommendation under the stated rollout rule.
Maple Cart wants to compare two checkout layouts. The experiment owner specifies assignment by eligible user, a planned 50/50 split, and completed orders per assigned eligible user as the decision metric. Those words define a population. Counting only people who clicked the new layout could exclude users affected by it and change the question.
You'll preserve assignments across sessions, log exposure separately, and keep conversion events idempotent. Repeated refreshes must not move a person between variants. The experiment also needs checks before anyone interprets a lift: assignment counts, missing exposure events, and a checkout-error guardrail. A result with uneven assignment or broken event collection can make one variant appear better even when the comparison is invalid. Your final recommendation will use supplied counts and a declared operating rule, without claiming statistical significance from a percentage alone.
Assignment key: experiment_id + user_id
Variant: stable hash bucket, fixed for this experiment version
Decision denominator: assigned eligible users
Exposure: a separate event, not a new assignment
Assignment defines the test population. Exposure records whether the assigned experience was displayed.