Text & documents
Numbers & maths
Data & formats
Security
Development & DevOps
Artificial Intelligence
Finance
Health & Wellness
Productivity
Games & Entertainment
Multimedia & design
Business
How to use
Analysing an experiment

Enter the visitors and conversions of each variant (A is the control). First the traffic split is checked (SRM): if it doesn't match the requested one, the result isn't reliable and you need to find the cause. Then you get the difference with its confidence interval, which tells you how much, not just whether. With little data or a rate at 0% or 100% Fisher's exact test is used, because the normal approximation declares winners that aren't.

What the p-value is (and isn't)

The p-value is the probability of seeing a difference like this or larger if there were really none. It is not the probability that B is better: the Bayesian part gives that, P(B > A). With more than two variants it is corrected for multiple comparisons (Holm or Benjamini-Hochberg): comparing several variants at once gives more chances to win by luck.

A/B TestAnalyse, plan and don't fool yourself by peeking
A/B testAnalyse with confidence intervals and an exact test, plan the sample size and see how much peeking early misleads you
VisitorsConversionsSplit % (optional)Rate
A Control (A)
12.00 %
B
14.00 %
Traffic split consistent with the requested one (SRM: p = 1.0000).
These data don't show any difference
Difference (B − A)+2.00 pp95% CI: -0.95 pp … +4.96 pp
Relative lift+16.7 %95% CI: -7.1 % … +46.5 %
p-value0.1836two-proportion z · z = 1.33
P(B better than A)90.8 %Bayesian, uniform priors

The p-value is NOT the probability that B is better: it is the probability of seeing a difference like this if there were none. And it only holds if you fixed the sample size beforehand and didn't stop on seeing a good result (“Peeking early” tab).