How this calculator works
It runs a two-tailed, two-proportion z-test - the standard method most A/B testing tools use. It compares the two conversion rates, accounts for sample size, and returns the probability that a difference this large would appear by chance if the variants were actually identical (the p-value).
z = (rate B - rate A) / sqrt( p x (1 - p) x (1/visitors A + 1/visitors B) )
where p = total conversions / total visitors
where p = total conversions / total visitors
How to read the result
- 95% confidence or higher - the conventional threshold to call a winner.
- 90 - 95% - promising, but keep running.
- Below 90% - you cannot tell the variants apart yet.
Common A/B testing mistakes
- Peeking and stopping early. Checking daily and stopping the moment you hit 95% inflates false positives. Decide the sample size up front.
- Tests shorter than a week. Weekday and weekend visitors behave differently - run at least one full business cycle.
- Too few conversions. Under ~100 conversions per variant, results swing wildly.
- Testing tiny changes on low traffic. Small sites should test bold changes (offer, headline, price) that can produce big uplifts.
Frequently asked questions
What confidence level should I use?
95% is the standard. For low-risk changes some teams accept 90%.
Is this one-tailed or two-tailed?
Two-tailed. It detects a difference in either direction, which is the safer default.