Free A/B Test Significance Calculator
You ran an A/B test, but can you trust the result? This free calculator checks in seconds whether the difference between variant A and variant B is statistically significant. Enter the visitor counts and conversions for both variants, choose your confidence level, and the calculator instantly tells you whether the result is reliable or whether you need more data.
Variant A (Baseline)
Variant B (Variation)
What does "statistically significant" mean?
Statistically significant means: the observed difference between variant A and variant B is unlikely to be random but it reflects a real effect.
The confidence level tells you how certain you want to be. At 95% confidence, this means: if you ran this test 100 times under identical conditions, you would see a result in this direction 95 times. Only 5 times would it be pure chance.
The most common mistake: declaring a winner as soon as variant B looks better without checking for significance. That leads to decisions based on noise instead of signal.
What this calculator computes: The two-tailed Z-test for two proportions. It compares the conversion rates of both variants and checks whether the difference exceeds your chosen significance threshold.
How do I read the result?
The calculator gives you three values:
Relative uplift: By how much does variant B outperform or underperform variant A? A relative uplift of +12% means variant B converts 12% better and not 12 percentage points better.
p-value: The probability that you would see this result (or one more extreme) by chance if there were truly no difference between the variants. A p-value below 0.05 corresponds to 95% confidence.
Result: Significant or not significant stated in plain language, with a clear recommendation on what to do next.
Important: "Not significant" does not mean variant B is worse. It means you don't have enough data yet to be sure.
When do I need more data?
If your test is not yet significant, there are two possible reasons:
- True null effect: There genuinely is no meaningful difference between the variants.
- Insufficient data: The effect exists, but you haven't seen enough visitors to measure it reliably.
As a rule of thumb: to reliably detect a 10% improvement in conversion rate at 95% confidence, you need roughly 4,000 visitors per variant. With 500 visitors per side, even real effects often won't show up yet.
Keep the test running until you either reach significance or hit your pre-defined minimum visitor count without peeking at the dashboard early and declaring a winner.
Frequently asked questions about the A/B test significance calculator
Document your experiments and learn from them
This calculator tells you whether your A/B test is significant. Blazeway tells you what to learn from it and makes sure that insight doesn't get lost. Hypothesis → test → result → insight → next hypothesis.
No cookie banner. No credit card. No overhead.