marketingmathcalc

A/B test calculator

Compare two conversion rates using a two-sided two-proportion z-test.

Last updated by Marketing Math Calc, published by AUSSIE-AI LTD

Task
Planning power
Confidence level (two-sided)
Solve for: tap the figure you want to find
from (rate B − rate A) ÷ pooled SE

Independent visitors assigned to A.

Visitors who converted, at most once each.

Independent visitors assigned to B.

Use the same conversion definition as A.

Planning rate before the change.

Smallest rate change worth detecting.

With 500 conversions from 10,000 visitors in A and 580 from 10,000 in B, B has a 16.0% relative lift and a p-value of 0.0123: significant at 95%. Enter your visitors and conversions to see the difference, uncertainty and calculation.

Set up the comparison

Count independent visitors randomly split into two separate groups. Each visitor belongs to one variant and has one binary outcome: converted or did not convert. Count converting visitors, not repeat purchases. Both groups need the same conversion definition and observation window. Unequal visitor counts are allowed.

Choose one primary metric and your confidence level before testing. This calculator tests for a difference in either direction, so a significant result can mean B is worse. Its null hypothesis is that the underlying conversion rates are equal, following the NIST two-proportion test.

Fix sample size in advance. Checking repeatedly and stopping when significance appears inflates false positives, as Evan Miller explains. This is a fixed-sample test, with no correction for repeated looks or multiple metrics.

Formula and working

Let pA = conversions A ÷ visitors A and pB = conversions B ÷ visitors B. Rates in these formulas are fractions, so 5% is 0.05.

Pooled p = (conversions A + conversions B) ÷ (visitors A + visitors B)

Pooled SE = sqrt(p × (1 − p) × (1/nA + 1/nB))

z = (pB − pA) ÷ pooled SE

Two-sided p-value = erfc(|z| ÷ sqrt(2))

The calculator uses Cody's published approximation for erfc. It compares the unrounded p-value with 0.10, 0.05 or 0.01 for 90%, 95% or 99% confidence. Small p-values display as <0.0001 rather than zero.

The interval uses a separate, unpooled standard error:

Unpooled SE = sqrt(pA × (1 − pA)/nA + pB × (1 − pB)/nB)

Difference interval = (pB − pA) ± z* × unpooled SE

Here z* is 1.644854, 1.959964 or 2.575829. Multiply endpoints by 100 for percentage points. Because the interval and test use different standard errors, their conclusions can differ near a cutoff. This is a normal-approximation interval, without continuity correction or clipping.

Read the example

A converts at 5.00%, B at 5.80%. The absolute difference is +0.80 percentage points, while relative lift is 0.80 ÷ 5.00 × 100 = +16.0%. The pooled rate is 0.054, z is 2.50, and the 95% difference interval is +0.17 to +1.43 percentage points. These are hypothetical inputs, not a benchmark.

A p-value is a tail probability under equal underlying rates and the test assumptions. It is not the probability that B is better. Statistical significance also does not measure whether the improvement is worth the cost. Use the conversion rate calculator for a single rate or its arithmetic increase.

When the approximation is weak

This tool flags any observed conversion or non-conversion cell below 10 in either arm. This is a screening rule, not a guarantee of accuracy. NIST describes an exact test for small samples. Do not rely on a normal approximation when the warning appears.

If both rates are 0% or both are 100%, the pooled standard error is zero. The calculator reports z and p as undefined, rather than claiming certainty. Relative lift is undefined whenever A's rate is zero; the percentage-point difference is still available.

Plan sample size before launch

Choose Plan sample size, enter a baseline rate and a target rate, then set confidence and power. Power is the chance of detecting the specified difference under the planning assumptions, not the chance that a finished test is correct. Use a meaningful absolute change rather than the largest lift you hope to see.

For a 5% baseline and 6% target, 95% two-sided confidence and 80% power, the equal-arm approximation is ceil((1.959964 + 0.841621)^2 × (0.05 × 0.95 + 0.06 × 0.94) / 0.01^2) = 8,155 visitors per arm. This is the unpooled planning approximation, without continuity correction; other planning methods may differ slightly. The rate difference is one percentage point. Keep the chosen horizon fixed, and allow each visitor's conversion window to finish before reading results. See running an A/B test for the complete workflow.

Questions

What does significant at 95% mean?

The two-sided p-value is below 0.05 under this model. It is evidence against equal underlying rates, conditional on the assumptions, not a 95% probability that B wins.

Can I stop when the result becomes significant?

Not with this fixed-sample test. Set your sample size beforehand and finish it. Stopping after repeated checks changes the false-positive rate; see Miller's explanation above.

Why can a negative lift be significant?

A two-sided test detects decreases as well as increases. A negative difference means B converted fewer visitors proportionally than A.

Is relative lift the same as percentage points?

No. Moving from 5% to 6% is a one-percentage-point increase and a 20% relative lift: (6 − 5) ÷ 5 × 100.

What should I do with a small-sample warning?

Treat the z-test and interval as unreliable. Consider an exact test such as the one described by NIST, and review your planned sample size before a future experiment.

Definitions we checked