Frequentist statistical power calculator
Find the statistical power of a fixed-horizon A/B test from the visitors in each variant, using a two-sided two-proportion z-test at your confidence level.
Statistical power
- Total across both variants
- 30,000
- Variant conversion rate
- 11%
How power is calculated
The calculator runs a two-sided two-proportion z-test. For a baseline rate p₁ and a variant rate p₂ that is p₁ plus the effect you enter, p₁ × (1 + lift) when the effect is relative, a test with n visitors in each variant has the power below, where p̄ is the average of the two rates, zα is the normal quantile for the confidence level, and Φ is the normal CDF.
power = Φ((|p₂ − p₁|·√n − zα·√(2p̄(1 − p̄))) ÷ √(p₁(1 − p₁) + p₂(1 − p₂)))
A test at 80% power still misses a real lift one time in five, so a result that falls short of significance at low power says little about whether the variant helps. The calculator also adds the small chance of a significant result in the opposite direction, which only shows at very small samples.