Home / Testing & CRO / A/B Test Significance Calculator

A/B Test Significance Calculator

Before you ship the "winner", check whether the difference is real or just random noise.

Variant A (control)
Variant B
Confidence B is different
-
Conv. rate A-
Conv. rate B-
Relative uplift-
p-value-

How this calculator works

It runs a two-tailed, two-proportion z-test - the standard method most A/B testing tools use. It compares the two conversion rates, accounts for sample size, and returns the probability that a difference this large would appear by chance if the variants were actually identical (the p-value).

z = (rate B - rate A) / sqrt( p x (1 - p) x (1/visitors A + 1/visitors B) )
where p = total conversions / total visitors

How to read the result

Common A/B testing mistakes

Worked example: is this winner real?

A store tests a new product page. Version A: 4,800 visitors, 144 orders (3.0%). Version B: 4,750 visitors, 171 orders (3.6%). That is a 20% relative lift.

Pooled conversion rate p = 315 / 9,550 = 3.3%. The standard error is the square root of 0.033 x 0.967 x (1/4,800 + 1/4,750), about 0.00366. The z-score is (0.036 - 0.030) / 0.00366, about 1.64, which is around 90% confidence (two-tailed). Promising, but below the usual 95% bar - keep the test running rather than shipping B today.

How many visitors do you need?

Sample size depends on your baseline conversion rate and the smallest lift you care about detecting. Roughly, at 95% confidence and 80% power:

Baseline conversionDetect 10% relative liftDetect 20% relative liftDetect 30% relative lift
1%~163,000 per variant~42,700 per variant~19,800 per variant
3%~53,200 per variant~13,900 per variant~6,400 per variant
5%~31,200 per variant~8,200 per variant~3,800 per variant
10%~14,800 per variant~3,800 per variant~1,800 per variant

The lesson for small sites: chasing 10% lifts on a 1% conversion rate needs more traffic than most sites have in a year. Test big changes that could plausibly move results by 20 to 30% or more. Our guide to A/B test duration explains how to plan this.

Statistical significance is not business significance

With enough traffic, a 0.5% lift can be statistically significant and still not worth the engineering time to maintain. Before testing, write down the minimum lift that would change a decision. After testing, check the confidence interval, not just whether you crossed 95%.

Frequently asked questions

What confidence level should I use?

95% is the standard. For low-risk changes some teams accept 90%.

Is this one-tailed or two-tailed?

Two-tailed. It detects a difference in either direction, which is the safer default.

What does 95% confidence actually mean?

If the two versions were truly identical, a difference at least this large would show up less than 5% of the time by random chance. It does not mean there is a 95% chance B is better.

Can I test more than two variants?

Yes, but each extra variant needs its own full sample and raises the chance of a false winner. Compare each variant to the control and use a stricter threshold, such as 99%, when testing many at once.

Embed this calculator on your site

Free for blogs, newsletters and client sites. Paste this HTML where you want the calculator - it resizes itself and works on any device. Please keep the credit line.

Get new calculators first

One short email when we add a tool or guide. No spam, unsubscribe any time.

Related tools

Paid adsCPM, CPC & CPA CalculatorAll your ad cost metrics - CPM, CPC, CTR, CPA, conversion rate - at once.SEO & trackingSERP Snippet Preview & Meta Length CheckerSee how your title and meta description look in Google - with pixel width.Paid adsROAS CalculatorReturn on ad spend plus your break-even ROAS from margin.

Related guides

GuideHow long to run an A/B testHow long to run an A/B test: sample size by conversion rate and expected lift, why to run at least one to two weeks, and the mistakes that create false winners.