Bayesian statistical power calculator
Find how often an A/B test reaches your probability-to-beat-control threshold from the visitors in each variant, when the lift you enter is real.
Statistical power
- Total across both variants
- 30,000
- Variant conversion rate
- 11%
How power is calculated
With a flat prior, the probability that the variant beats control is the normal CDF of the observed difference divided by its standard error. The power is how often that probability reaches the one you set when the true lift is the one you entered, where zP is the normal quantile for that probability and Φ is the normal CDF.
power = Φ(|p₂ − p₁|·√n ÷ √(p₁(1 − p₁) + p₂(1 − p₂)) − zP)
At the same visitors the Bayesian figure comes out higher than the frequentist one because it asks for one-sided evidence that the variant is better, where the frequentist test also guards against the variant being worse.