Quick summary
Summarize this blog with AI
A strong A/B testing answer connects the product decision, assignment, metric, uncertainty, and rollout rule. This guide uses a fictional conversion experiment with one binary outcome per independently assigned user. Its calculations are illustrative, not a substitute for designing an experiment around your actual product.
Design before looking at outcomes
Suppose a team wants to change onboarding. Define conversion as completing a first project within seven days of assignment. Specify eligibility, stable user-level assignment, the event definition, timezone, full observation window, and how missing events will be investigated. Analyze fully observed cohorts; a user assigned yesterday has not completed a seven-day window.
Before launch, choose a primary metric, practical effect of interest, power and sample-size plan, minimum duration covering relevant cycles, and decision rule. Record guardrails such as errors and support burden. A sample-size calculation depends on the baseline, effect size, variance, assignment unit, desired power, and analysis. There is no universal rule that two weeks or 1,000 users is sufficient.
Validate the experiment before interpreting lift
Check assignment counts against the intended allocation, event completeness, duplicate users, and leakage between variants. A sample ratio mismatch can signal an assignment or logging problem; investigate it before trusting effects. Microsoft’s experimentation guidance explains this diagnostic.
Keep the analysis population consistent with the plan. Selecting only people who clicked a treatment-specific feature can introduce post-treatment selection bias. If assignment is by household, team, or market, use an analysis that respects that unit and dependence. Treating repeated sessions as independent users understates uncertainty.
Calculate a complete conversion comparison
Assume a valid experiment has finished its planned collection and follow-up. Control has 100 conversions among 1,000 users; treatment has 120 among 1,000. The observed rates are 10% and 12%. The absolute difference is 2 percentage points; the relative increase is 20%. Those are different ways of describing the same point estimate.
The following Python standard-library code calculates a two-sided pooled normal test for equal proportions and an unpooled approximate 95% interval for the difference. The formulas follow the pooled proportions test and Wald difference interval described by NIST. Both are large-sample approximations. Expected successes and failures are comfortably large in this example; sparse data or complex assignment needs a different method.
from math import erfc, sqrt
from statistics import NormalDist
control_success, control_n = 100, 1000
treatment_success, treatment_n = 120, 1000
pc = control_success / control_n
pt = treatment_success / treatment_n
difference = pt - pc
pooled = (control_success + treatment_success) / (control_n + treatment_n)
null_se = sqrt(pooled * (1-pooled) * (1/control_n + 1/treatment_n))
z = difference / null_se
p_value = erfc(abs(z) / sqrt(2))
interval_se = sqrt(pc*(1-pc)/control_n + pt*(1-pt)/treatment_n)
margin = NormalDist().inv_cdf(.975) * interval_se
print(f"absolute lift: {100*difference:.2f} percentage points")
print(f"relative lift: {difference/pc:.2%}")
print(f"two-sided p-value: {p_value:.4f}")
print(f"95% interval: [{100*(difference-margin):.2f}, "
f"{100*(difference+margin):.2f}] percentage points")
The p-value is about 0.153 and the interval runs approximately from −0.74 to 4.74 percentage points. Under a predeclared two-sided 5% test, this result does not establish a nonzero effect. It also does not prove the variants are equivalent or rule out a worthwhile improvement.
Connect the result to a decision
Report the estimate and interval alongside guardrails and validity checks. If the practical threshold was a 1-percentage-point improvement, this interval does not establish that threshold either. Do not relabel the analysis as one-sided after observing a favorable direction.
A decision may be to keep the current experience, gather a new adequately planned experiment, or choose based on other documented costs and benefits. State the uncertainty. If the original sample-size plan was too small for the practical effect, explain that limitation instead of treating a nonsignificant result as “no impact.”
Handle repeated looks and many metrics deliberately
A fixed-horizon p-value is not permission to inspect every day and stop the first time it crosses 0.05. Use a prespecified fixed analysis or an appropriate sequential design with its own decision rules. Safety monitoring may require stopping for harm, but that does not turn an opportunistic efficacy claim into a valid fixed-horizon test.
Predefine confirmatory metrics and any multiplicity strategy. Label unplanned segment results as exploratory. Investigate novelty effects, delayed outcomes, and interactions with concurrent experiments where relevant. Neither a testing platform nor a significant number removes the need to understand assignment and measurement.
Give an interview-ready recommendation
“The observed conversion difference is +2 percentage points, with an approximate 95% interval of −0.74 to +4.74. After validating assignment and measurement, I would report that the planned test is inconclusive about a positive effect and does not establish equivalence. I would compare this uncertainty and the guardrails with the prespecified decision criteria.”
For practice, change the conversion counts and predict the direction of the interval before rerunning. Then explain why the same calculation would be inappropriate for revenue with heavy tails or for users clustered within assigned organizations.
