What you'll learn
Quick Answer
An A/B test randomly splits users into two groups, exposes them to different versions, and checks whether the difference in an outcome, like conversion rate, is bigger than what random chance alone would produce. A two-proportion z-test converts that difference into a p-value; 'statistically significant' means the observed gap would be unlikely if there were actually no real difference — it does not mean the gap is large, important, or even real, and a p-value just above or below 0.05 is not a hard line between true and false.
What an A/B Test Is Actually Checking
An A/B test randomly splits users into two (or more) groups and shows each group a different version of something — a button color, a pricing page, a recommendation algorithm — then measures an outcome, like whether the user converted. Random assignment is what makes the comparison fair: done correctly, the two groups should be statistically identical in every way except which version they happened to see, so any difference you observe in the outcome is attributable to the version, not to some other underlying difference between the groups.
The statistical question underneath is precise: assume the null hypothesis (H0) that both versions actually have the exact same true conversion rate, and ask how surprising the difference you actually observed would be if that assumption were true. If the observed gap would be very unlikely under H0, that's evidence against H0 — evidence the versions really do differ.
Because 'did this specific user convert' is a yes/no outcome, the group-level conversion rate is a proportion, which is exactly what the two-proportion z-test is built to compare.
The Two-Proportion Z-Test Formula
The test pools both groups together under the assumption that H0 is true — a single shared 'true' conversion rate — because if the null hypothesis holds, there's no reason to treat the two groups' data as coming from different distributions when estimating what that shared rate probably is: p_pool = (conversions_A + conversions_B) / (n_A + n_B).
From that pooled rate, the standard error of the difference between the two groups' observed rates is SE = sqrt(p_pool × (1 - p_pool) × (1/n_A + 1/n_B)). The z-score is then simply how many standard errors apart the two observed rates are: z = (p_B - p_A) / SE. A larger z means the gap is less likely to be pure sampling noise.
Running It for Real in Python
Dataset: Variant A converted 480 out of 10,000 users (4.80%); Variant B converted 540 out of 10,000 (5.40%) — a 0.60 percentage-point absolute gap, or 12.5% better in relative terms.
p_pool = (conv_a + conv_b) / (n_a + n_b)
se_pool = math.sqrt(p_pool * (1 - p_pool) * (1 / n_a + 1 / n_b))
z = (p_b - p_a) / se_pool
p_value = 2 * (1 - stats.norm.cdf(abs(z)))Real output from actually running this: p_pool = 0.05100, SE = 0.003111, z = 1.9285, two-tailed p-value = 0.05379.
Before trusting a number that lands this close to a meaningful threshold, it's worth cross-checking it a completely different way. Running a chi-square test of independence on the same data as a 2x2 contingency table (converted/not-converted by A/B), via scipy.stats.chi2_contingency, gives chi2 = 3.7191 and p = 0.05379 — and chi2 here should mathematically equal z^2 for a two-group comparison like this one: 1.9285^2 = 3.7191. Both independent calculations agree exactly, which is the kind of cross-check worth doing any time a result is close enough to a decision boundary to matter.
Reading the Actual Result
p = 0.05379 sits just above the conventional 0.05 cutoff, meaning this result is not statistically significant at the standard 5% level, even though Variant B genuinely converted better in the raw numbers: 5.40% versus 4.80%, a real 12.5% relative lift.
This is a realistic, common outcome, not an edge case constructed for this article — a huge number of real A/B tests land close to the boundary rather than comfortably on one side of it, and it's exactly the situation where misreading the result does the most damage. Seeing 'not significant' and concluding 'B doesn't actually work' is a different, and often wrong, claim from the honest one: 'we don't yet have enough data to distinguish this effect from noise with confidence.'
What the P-Value Does Not Mean
p = 0.05379 means: if A and B truly had identical conversion rates, there is a 5.4% chance that random sampling alone would produce a gap this large or larger just by chance. It does not mean 'there's a 94.6% chance B is really better,' and it does not mean 'there's a 5.4% chance B isn't really better.' Both of those statements invert what's actually a conditional probability: the p-value is P(this data | H0 is true), not P(H0 is true | this data) — and confusing the two is one of the most common statistical mistakes in the industry, not a minor technicality.
It's also worth separating statistical significance from practical significance. With a large enough sample, a genuinely trivial difference (0.01 percentage points) can become statistically significant. With a small enough sample, a genuinely large, real difference can fail to reach significance, which is closer to what's happening here: 10,000 users per arm is often not quite enough to reliably distinguish a 0.6 percentage-point absolute gap on top of a roughly 5% baseline rate from noise.
Sample Size, Power, and When to Actually Stop
Statistical power is the probability a test will actually detect a real effect of a given size, if that effect genuinely exists. A test that's under-powered for the true effect size will frequently produce exactly this kind of borderline, inconclusive result — not because there's no real difference, but because the sample size wasn't large enough to reliably tell the difference from noise, even when one is genuinely there.
The honest fix for a borderline result like this one is not to relax the threshold or squint at the number until it looks favorable — it's to decide the required sample size, based on the baseline rate and the smallest effect size worth caring about (the minimum detectable effect), before the test starts. If the test ends borderline, treat that as a real, valid answer: either run longer with a pre-committed larger sample, or accept that no reliable difference was detected at the sample size actually used.
One specific trap deserves its own warning: 'peeking,' checking the p-value every day and stopping the moment it first crosses 0.05, inflates the true false-positive rate well above the nominal 5%, because every extra look is another chance for noise to briefly cross the threshold before reverting. A test evaluated once, at a sample size decided in advance, is what the 5% figure actually promises; a test checked daily until it happens to look good is a different, much less reliable procedure wearing the same label.