marketingmathcalc

Running an A/B test from plan to decision

Plan the effect you need to detect before collecting results. A 5% baseline and 6% target require about 8,155 visitors per arm at 95% two-sided confidence and 80% power using this calculator's approximation. A completed test then needs a separate interpretation of observed lift, uncertainty and practical value before you decide what to ship.

Last updated by Marketing Math Calc, published by AUSSIE-AI LTD

Write the decision before designing the variant

Suppose a team wants to simplify a signup page. The experiment compares the existing page, A, with a revised page, B. Its primary outcome is whether an assigned visitor signs up within the agreed observation window. Count each visitor once. Repeat visits and repeat submissions must not create additional independent observations.

Describe what the team would do if the new page helps, harms or produces an inconclusive result. A statistically detectable improvement might still be too small to justify maintenance work. Conversely, a simpler page could have operational benefits that deserve a separate evaluation. Record these considerations before seeing which version appears ahead so that the decision criteria do not move with the result.

Define who is eligible and how assignment persists across visits. A visitor who switches between pages breaks the intended comparison. Choose a consistent conversion window so people assigned near the end have the same opportunity to convert as people assigned at the start. The test cannot compensate for different measurement rules in the two groups.

Plan the visitors required

Use the planning mode of the A/B test calculator. Enter a 5% baseline, a 6% target, 95% confidence and 80% power. The absolute difference is one percentage point. This is a hypothetical planning effect, not a prediction that the redesigned page will achieve it.

The equal-arm normal approximation uses the sum of the confidence and power critical values, squared, multiplied by the two rate variances and divided by the squared difference. Here it is ceil((1.959964 + 0.841621)^2 × (0.05 × 0.95 + 0.06 × 0.94) / 0.01^2), giving 8,155 visitors per arm. Rounding upward avoids planning less than the calculated requirement.

Power describes detection probability for the specified planning effect under the model. It is not the probability that your eventual conclusion will be correct. A smaller meaningful effect requires a larger experiment. If the planned traffic cannot support that effect, reconsider the scope before launch rather than waiting until an underpowered result disappoints you.

Fix the horizon and protect the comparison

Choose the final allocation and stopping rule in advance. For the reading example below, suppose the team schedules 10,000 visitors per arm, exceeding the calculated minimum, with complete conversion windows before analysis. That decision happens before observing results. It is not a response to a promising interim p-value.

Randomly assign independent eligible visitors, keep both variants running concurrently, and log implementation changes. Avoid modifying the offer or the conversion event halfway through. Monitor technical failures, but distinguish operational monitoring from repeatedly testing for significance. Evan Miller's fixed-horizon discussion explains why stopping whenever a conventional p-value looks favorable changes the false-positive behavior.

Inspect allocation and tracking before interpreting the outcome. Missing events in one variant, bots in only one traffic source or a broken page can produce a numerical difference without answering the intended product question. Document exclusions consistently. If a fault invalidates the comparison, repair it and design a fresh run instead of trying many filters until the desired result appears.

Read the completed result

A records 500 conversions from 10,000 visitors, or 5.0%. B records 580 from 10,000, or 5.8%. The absolute lift is 0.8 percentage points, and relative lift is 16%. The conversion-rate calculator can verify the rates separately if those two ways of describing change are unfamiliar.

The combined conversion proportion is 0.054. Under the equal-rate null hypothesis, the pooled standard error is sqrt(0.054 × 0.946 × (1/10000 + 1/10000)) = 0.0031964. Dividing the observed difference, 0.008, by that error gives z = 2.50. The two-sided tail probability is approximately 0.012, displayed more precisely as 0.0123.

At the chosen confidence level, this result is statistically significant. That means the observed difference would be relatively unusual under equal underlying rates and the test assumptions. It does not give the probability that B is better, prove the change will persist, or identify why the difference happened. Randomization, measurement and the stopping rule remain part of the interpretation.

Use the interval to judge practical value

The confidence interval uses the unpooled standard error, calculated from each arm's observed rate. For this result it is 0.0031959. The 95% interval for B minus A is +0.17 to +1.43 percentage points. These endpoints show the uncertainty in the estimated absolute change rather than presenting the observed lift as a guaranteed outcome.

Compare that range with the decision written before launch. Would the smaller improvement still justify the implementation cost? Are there unresolved effects on customer quality or later outcomes? Keep those questions separate from the significance label. A positive signup result is evidence about signup behavior under this experiment, not automatic evidence of higher long-term profit.

The test and interval use different standard errors, so they can disagree near a threshold. Both rely on a normal approximation. The calculator warns when any observed conversion or non-conversion cell contains fewer than ten observations. Large total traffic alone does not ensure enough events for a rare outcome.

Contrast an inconclusive finish

Keep A unchanged, but suppose B had 540 conversions from the same number of visitors. B's rate would be 5.4%, and the observed difference would be 0.004 as a fraction. The pooled rate becomes 0.052, giving SE = sqrt(0.052 × 0.948 × 0.0002) = 0.0031399. The resulting z is 1.27, with p approximately 0.20.

This result is not significant at the planned threshold. It does not prove the pages are equivalent or establish that the change has no effect. The observed improvement is simply not sufficiently separated from sampling noise under this test. Report the completed result and its uncertainty rather than continuing until the label changes.

Make the next decision explicit

For the first outcome, weigh the evidence alongside implementation costs and any prespecified quality checks. For the contrast, retain the current page or design another experiment if the question remains worthwhile. A new run needs a fresh plan; it should not silently become an extension of a finished test.

Archive the assignment rules, inputs, stopping rule and final interpretation together. That record lets the next team understand what was actually tested. The useful deliverable is a decision with an evidence trail, including an honest inconclusive result when the data cannot distinguish the alternatives clearly enough.

Sources