How this calculator works
It runs a two-tailed, two-proportion z-test - the standard method most A/B testing tools use. It compares the two conversion rates, accounts for sample size, and returns the probability that a difference this large would appear by chance if the variants were actually identical (the p-value).
where p = total conversions / total visitors
How to read the result
- 95% confidence or higher - the conventional threshold to call a winner.
- 90 - 95% - promising, but keep running.
- Below 90% - you cannot tell the variants apart yet.
Common A/B testing mistakes
- Peeking and stopping early. Checking daily and stopping the moment you hit 95% inflates false positives. Decide the sample size up front.
- Tests shorter than a week. Weekday and weekend visitors behave differently - run at least one full business cycle.
- Too few conversions. Under ~100 conversions per variant, results swing wildly.
- Testing tiny changes on low traffic. Small sites should test bold changes (offer, headline, price) that can produce big uplifts.
Worked example: is this winner real?
A store tests a new product page. Version A: 4,800 visitors, 144 orders (3.0%). Version B: 4,750 visitors, 171 orders (3.6%). That is a 20% relative lift.
Pooled conversion rate p = 315 / 9,550 = 3.3%. The standard error is the square root of 0.033 x 0.967 x (1/4,800 + 1/4,750), about 0.00366. The z-score is (0.036 - 0.030) / 0.00366, about 1.64, which is around 90% confidence (two-tailed). Promising, but below the usual 95% bar - keep the test running rather than shipping B today.
How many visitors do you need?
Sample size depends on your baseline conversion rate and the smallest lift you care about detecting. Roughly, at 95% confidence and 80% power:
| Baseline conversion | Detect 10% relative lift | Detect 20% relative lift | Detect 30% relative lift |
|---|---|---|---|
| 1% | ~163,000 per variant | ~42,700 per variant | ~19,800 per variant |
| 3% | ~53,200 per variant | ~13,900 per variant | ~6,400 per variant |
| 5% | ~31,200 per variant | ~8,200 per variant | ~3,800 per variant |
| 10% | ~14,800 per variant | ~3,800 per variant | ~1,800 per variant |
The lesson for small sites: chasing 10% lifts on a 1% conversion rate needs more traffic than most sites have in a year. Test big changes that could plausibly move results by 20 to 30% or more. Our guide to A/B test duration explains how to plan this.
Statistical significance is not business significance
With enough traffic, a 0.5% lift can be statistically significant and still not worth the engineering time to maintain. Before testing, write down the minimum lift that would change a decision. After testing, check the confidence interval, not just whether you crossed 95%.
Frequently asked questions
What confidence level should I use?
95% is the standard. For low-risk changes some teams accept 90%.
Is this one-tailed or two-tailed?
Two-tailed. It detects a difference in either direction, which is the safer default.
What does 95% confidence actually mean?
If the two versions were truly identical, a difference at least this large would show up less than 5% of the time by random chance. It does not mean there is a 95% chance B is better.
Can I test more than two variants?
Yes, but each extra variant needs its own full sample and raises the chance of a false winner. Compare each variant to the control and use a stricter threshold, such as 99%, when testing many at once.