A/B test significance calculator

What does statistical significance mean in an A/B test?

An A/B test shows two versions of a page to two random groups and compares their conversion rates. Because each group is a sample, the rates would differ a little even if both versions were identical. Statistical significance asks: if the versions really performed the same, how surprising would a gap this large be?

That surprise is the p-value. A p-value of 0.018 means that, if there were no real difference, you’d see a gap at least this large about 1.8% of the time. At a 95% confidence level you accept results with p below 0.05, so you call it significant — knowing that roughly 1 in 20 such calls on truly identical variants will be a false alarm.

The formula: two-proportion z-test

The normal approximation behind this test works well once each variant has at least a few dozen conversions and non-conversions. With single-digit conversion counts, treat any result as directional and keep collecting data.

Worked example

A pricing page test sends 10,000 visitors to each variant. The control (A) gets 300 signups, the new layout (B) gets 360.

1. Rates: pA = 300 ÷ 10,000 = 3.0%, pB = 360 ÷ 10,000 = 3.6%. Relative uplift = 0.6 ÷ 3.0 = +20%.
2. Pooled rate: p = 660 ÷ 20,000 = 3.3%.
3. Standard error: √(0.033 × 0.967 × (1/10,000 + 1/10,000)) ≈ 0.002526.
4. z = 0.006 ÷ 0.002526 ≈ 2.375.
5. Two-sided p-value = 2 × (1 − Φ(2.375)) ≈ 0.0175 — below 0.05, so significant at 95% confidence.
6. 95% confidence interval for the difference: about +0.10 to +1.10 percentage points. B is very likely better, but the true lift could be anywhere from a few percent to over 35%.

Now change B to 340 conversions in the calculator: p rises to about 0.11 and the result is no longer significant. Small differences in conversions move the verdict a lot at this traffic level, which is why the test length should be fixed up front with the sample size calculator.

How often do A/B tests win? Benchmarks

Expect most ideas not to win. In “Online Experimentation at Microsoft” (Kohavi et al., 2009), the authors report that only about one third of the ideas tested improved the metric they were designed to improve; the rest were flat or negative. Win rates vary by team and product, but that base rate is a useful corrective: a test result that seems too good is more often a statistical fluke or a tracking bug than a breakthrough.

Common A/B testing mistakes

• Peeking and stopping early. Checking every day and stopping the first time p dips below 0.05 makes false positives far more likely than 5%. Fix the sample size first, then evaluate once.
• Testing many variants or metrics and reporting the winner. With 5 variants vs control at α = 0.05, the chance that at least one looks significant by luck is about 23%. Correct for it (e.g. Bonferroni: α ÷ number of comparisons).
• Choosing one-sided after seeing the data. A one-sided test halves the p-value; pick it before the test only if you’d never ship a version that’s worse.
• Ignoring sample ratio mismatch. If you split 50/50 but got 10,000 vs 9,200 visitors, the assignment or tracking is broken and the result can’t be trusted.
• Counting bots. Crawlers rarely convert and can land unevenly in variants; filter them before counting visitors.
• Stopping before a full week. Weekday and weekend visitors behave differently; run whole weeks.

How to get A/B test numbers from VisitTrack

VisitTrack isn’t an experimentation platform — it doesn’t split traffic for you. But if your testing tool, feature flag or own code assigns the variant, it gives you clean visitor and conversion counts for this calculator, with bots already removed.

1. Serve each variant on its own URL (e.g. /pricing and /pricing-b), or fire an event per variant view such as window.visitrack("pricing_view_b") (custom events).
2. Create a funnel per variant — variant view → signup. The Funnels tab shows entered, completed and the conversion rate for each.
3. Copy the entered and completed numbers into the calculator above.
4. If the test is about revenue rather than signups, connect your payment provider (revenue attribution) and compare revenue per visitor by landing page, not just conversion rate.