Mid standard

• Connect the assumptions, method and result: I use purchases per assigned user as the primary intent-to-treat outcome because excluding nonbuyers changes the randomized population. I define a refund guardrail with a common follow-up window and check logging and assignment first.
• Test the consequential edge case: I check random assignment, sample ratio, exposure logging and whether users can affect each other because those conditions support causal interpretation. I estimate uncertainty for the prespecified outcome and avoid stopping at the first favorable result.
• Explain an owned validation or operating check: I would hold broad rollout while checking refund maturity and the guardrail because higher purchases alone do not establish better customer outcomes. I report both effects and uncertainty, then agree a decision rule rather than declaring a win from the headline rate.

Lessons to prepare

Mid standard

• Define mature eligible-user cohorts, compare device mix and verify events before proposing a controlled test.
• I compare mobile and desktop cohorts separately because changing traffic mix can change the aggregate rate, and I reconcile raw events before trusting the instrumentation. Matching reporting windows helps expose missing completions near the cutoff.
• I use random assignment at user level with a clear guardrail and a stopping rule because users switching flows would contaminate a controlled experiment. I set the sample size from a worthwhile effect and allow every assigned cohort its completion window.

Lessons to prepare

Mid standard

• Map upgrades to stable customer identities, exclude immature observations and distinguish a retention rate from a count.
• I map each upgrade and duplicate account back to its stable customer identity before counting because otherwise the same person can become both a new and returning customer. I check unmatched mappings and distinguish a logging outage from zero activity.
• I compare customer groups with a control or random experiment because selection into the plan may correlate with prior activity before the plan changed anything. I retain assignment for intent-to-treat analysis if some customers decline the plan.

Lessons to prepare

Mid standard

• Use baseline, minimum detectable effect, significance and power in a sample-size calculation, then convert eligible traffic to duration.
• I divide the required assigned users by eligible daily traffic and then allow the seven-day outcome window because enrollment ending does not make the newest outcomes mature. I cover the weekly operating cycle and check assignment balance before interpreting results.
• I keep the prespecified fixed-horizon decision unless a valid sequential design was chosen because repeatedly stopping at significance increases false positives. I monitor a separately defined harm guardrail and can stop for safety without declaring an early efficacy win.

Lessons to prepare

Mid standard

• Separate enrollment week from time since exposure, preserve assignment and compare mature weekly effects with uncertainty.
• Daily peeking followed by stopping at the first significant result inflates false-positive risk because the fixed-horizon test assumes its stated analysis plan. I use the prespecified final analysis or a valid sequential method and report all planned outcomes.
• I would hold broad rollout until the sustained effect and quality guardrail are clear because an early novelty gain does not establish lasting value. A fresh prespecified cohort or longer mature follow-up can test persistence without rewriting the success rule after seeing results.

Lessons to prepare

Mid standard

• Hold ticket mix constant, check open-ticket censoring and define the response event before attributing a change.
• Using the old 80-to-20 mix with the new segment times gives 4.8 hours because 80 times 3 plus 20 times 12 is 480. Both segments slowed, so I separate that deterioration from the larger share of simple tickets.
• I recommend investigating response within each ticket type because the lower aggregate average reflects mix rather than faster service. I would publish segment counts and tail latency alongside the average, then test routing with comparable arrivals and a backlog guardrail.

Lessons to prepare