The sample size formula
This calculator uses the standard two-sided test for the difference between two proportions:
where p1 is the baseline conversion rate, p2 is the rate you want to be able to detect (baseline x (1 + lift)), p is their average, a is 1 minus the confidence level, and z(...) is the standard normal quantile - 1.96 for 95% confidence and 0.84 for 80% power.
Worked example
A product page converts at 3% and you want to reliably detect a 20% relative lift - from 3% to 3.6%. At 95% confidence and 80% power the formula gives 13,914 visitors per variant, or 27,828 for an A/B test. With 1,000 visitors a day the test needs 28 days. Each variant will see roughly 417 conversions over that time.
Why the lift you choose matters so much
Sample size grows with the square of the smaller difference. Halve the lift you want to detect and you need roughly four times the traffic:
| Baseline | Lift to detect | Visitors per variant (95% / 80%) |
|---|---|---|
| 3% | 10% | ~53,200 |
| 3% | 20% | ~13,900 |
| 3% | 30% | ~6,500 |
| 10% | 10% | ~14,800 |
| 10% | 20% | ~3,800 |
That is why low-traffic sites should test bold changes - a new offer, a new page layout, a different price - rather than button colours. Small tweaks produce small lifts that need huge samples to prove.
Rules for a clean test
- Fix the sample size before you start, and stop only when you reach it. Checking daily and stopping at the first "significant" result inflates false positives dramatically.
- Run whole weeks. Weekday and weekend visitors behave differently; a test should cover at least one, preferably two, full weekly cycles.
- Split traffic evenly and randomly, and keep returning visitors in the same variant.
- Pick one primary metric before launch. Looking at ten metrics guarantees one of them will look significant by chance.
- Adding variants costs traffic. An A/B/C/D test needs twice the total visitors of an A/B test.
When the test finishes, check the result with the A/B test significance calculator, and translate the lift into money with the conversion rate calculator.
Frequently asked questions
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you care about. A 3% baseline and a 20% relative lift needs about 13,900 visitors per variant at 95% confidence and 80% power.
What is statistical power?
The probability that the test detects a real effect of the size you specified. 80% power means that if the variant truly is 20% better, the test will show a significant result 8 times out of 10.
Can I stop a test early if it is already significant?
Not with this fixed-sample method. Repeatedly checking and stopping at the first significant result greatly increases the chance of a false winner. Decide the sample size up front and wait for it, or use a sequential testing method designed for peeking.