Zway.ai
Book a demo

Free tool

Is that lift real, or is it noise

Enter visitors and conversions for both variants and this runs a two proportion z test, reporting conversion rates, relative lift, and the p value. Significance at ninety five percent means the difference is unlikely to be noise. It does not mean the difference is large enough to be worth shipping.

Last reviewed 27 August 2026

Result

Conversion rate, A

5.00%

250 of 5,000

Conversion rate, B

5.80%

290 of 5,000

Relative lift

+16.0%

Absolute difference 0.80%

p value, two tailed

0.0768

z of 1.770 against a threshold of 0.05

The two variants

The two variants
VariantVisitorsConversionsRate
A5,0002505.00%
B5,0002905.80%

This does not reach 95 percent confidence. A p value of 0.0768 means a difference this large would turn up reasonably often even if the two variants were identical. Either keep running or accept that you cannot separate them.

How this works

What the numbers mean.

  • 01Both variants are treated as proportions. The test pools them into a single rate, computes the standard error of the difference under the assumption that the two are the same, and expresses the observed gap as a z score in standard errors.
  • 02The p value is the two tailed area beyond that z score in a standard normal distribution. It is computed here with the Abramowitz and Stegun 7.1.26 approximation to the error function, which is accurate to about seven decimal places.
  • 03The confidence interval uses the unpooled standard error, because once you accept the variants differ, pooling them is no longer the right assumption.

Assumptions and limits

  • Statistical significance is not practical significance. A tiny lift can be significant with enough traffic and still not be worth the work of shipping it.
  • Checking results before the planned sample size is reached inflates false positives badly. Every look at the data is another chance to see noise that happens to cross the threshold.
  • This assumes visitors were split randomly and independently, and that you decided what to measure before you started. Picking the winning metric afterwards makes the p value meaningless.

Questions about this tool

What does a p value of 0.05 actually mean?
It means that if the two variants were genuinely identical, you would see a difference at least this large about five percent of the time from random variation alone. It is not the probability that B is better, and it is not the probability that you are right. That distinction matters because the second reading makes the evidence sound far stronger than it is.
Why is peeking at results a problem?
Because a running test wanders. Given enough looks, a completely null test will cross the ninety five percent line at some point purely by chance, and stopping at that moment converts noise into a decision. Set the sample size before you start, then look once. If you want to monitor continuously, you need a sequential testing method rather than this one.
My result is significant but the lift is tiny. Should I ship it?
Probably not on its own. Significance answers whether the difference is real, not whether it is worth having. A one percent relative lift on a small base can be real and still be worth less than the engineering time to maintain the change. Decide the minimum lift you care about before running the test, not after.
Is a two tailed test right for my case?
It is the safer default and it is what this tool uses. A one tailed test asks only whether B beats A and gets significance more easily, which is fine if you genuinely would not act on a negative result. Most teams would act on a large drop, which makes the two tailed test the honest choice.
What if I am comparing more than two variants?
Run each comparison separately and tighten your threshold, because testing several variants against a control multiplies the chance of a false positive. With three variants against one control at ninety five percent, the chance of at least one spurious winner is meaningfully above five percent.

Related

This tool is free and there is nothing to sign up for. If you would rather have the work done than calculate it, that is what Zway does.

All toolsBook a demo