With 500 conversions from 10,000 visitors in A and 580 from 10,000 in B, B has a 16.0% relative lift and a p-value of 0.0123: significant at 95%. Enter your visitors and conversions to see the difference, uncertainty and calculation.
Set up the comparison
Count independent visitors randomly split into two separate groups. Each visitor belongs to one variant and has one binary outcome: converted or did not convert. Count converting visitors, not repeat purchases. Both groups need the same conversion definition and observation window. Unequal visitor counts are allowed.
Choose one primary metric and your confidence level before testing. This calculator tests for a difference in either direction, so a significant result can mean B is worse. Its null hypothesis is that the underlying conversion rates are equal, following the NIST two-proportion test.
Fix sample size in advance. Checking repeatedly and stopping when significance appears inflates false positives, as Evan Miller explains. This is a fixed-sample test, with no correction for repeated looks or multiple metrics.
Formula and working
Let pA = conversions A ÷ visitors A and pB = conversions B ÷ visitors B. Rates in these formulas are fractions, so 5% is 0.05.
Pooled p = (conversions A + conversions B) ÷ (visitors A + visitors B)
Pooled SE = sqrt(p × (1 − p) × (1/nA + 1/nB))
z = (pB − pA) ÷ pooled SE
Two-sided p-value = erfc(|z| ÷ sqrt(2))
The calculator uses Cody's published approximation for erfc. It compares the unrounded p-value with 0.10, 0.05 or 0.01 for 90%, 95% or 99% confidence. Small p-values display as <0.0001 rather than zero.
The interval uses a separate, unpooled standard error:
Unpooled SE = sqrt(pA × (1 − pA)/nA + pB × (1 − pB)/nB)
Difference interval = (pB − pA) ± z* × unpooled SE
Here z* is 1.644854, 1.959964 or 2.575829. Multiply endpoints by 100 for percentage points. Because the interval and test use different standard errors, their conclusions can differ near a cutoff. This is a normal-approximation interval, without continuity correction or clipping.
Read the example
A converts at 5.00%, B at 5.80%. The absolute difference is +0.80 percentage points, while relative lift is 0.80 ÷ 5.00 × 100 = +16.0%. The pooled rate is 0.054, z is 2.50, and the 95% difference interval is +0.17 to +1.43 percentage points. These are hypothetical inputs, not a benchmark.
A p-value is a tail probability under equal underlying rates and the test assumptions. It is not the probability that B is better. Statistical significance also does not measure whether the improvement is worth the cost. Use the conversion rate calculator for a single rate or its arithmetic increase.
When the approximation is weak
This tool flags any observed conversion or non-conversion cell below 10 in either arm. This is a screening rule, not a guarantee of accuracy. NIST describes an exact test for small samples. Do not rely on a normal approximation when the warning appears.
If both rates are 0% or both are 100%, the pooled standard error is zero. The calculator reports z and p as undefined, rather than claiming certainty. Relative lift is undefined whenever A's rate is zero; the percentage-point difference is still available.
Plan sample size before launch
Choose Plan sample size, enter a baseline rate and a target rate, then set confidence and power. Power is the chance of detecting the specified difference under the planning assumptions, not the chance that a finished test is correct. Use a meaningful absolute change rather than the largest lift you hope to see.
For a 5% baseline and 6% target, 95% two-sided confidence and 80% power, the equal-arm approximation is ceil((1.959964 + 0.841621)^2 × (0.05 × 0.95 + 0.06 × 0.94) / 0.01^2) = 8,155 visitors per arm. This is the unpooled planning approximation, without continuity correction; other planning methods may differ slightly. The rate difference is one percentage point. Keep the chosen horizon fixed, and allow each visitor's conversion window to finish before reading results. See running an A/B test for the complete workflow.