Your files stay on your device

A/B test significance calculator

Enter conversions or means for two groups and get a significance verdict with p-value, confidence interval, effect size, and achieved power.

Enter both arms to test the difference.

The proportion test is the pooled two-proportion z-test with an unpooled confidence interval; the means test is Welch's t-test, which does not assume equal variances. Both are fixed-horizon readings: decide at the planned end, because stopping early on a promising p-value inflates the false-positive rate. For tiny conversion counts the normal approximation gets shaky, and the tool says so instead of guessing. Power is a normal approximation against the observed standard error, not a noncentral-t computation.
Options

More fake data tools

- Cohort retention analyzer (/data/cohort-retention-analyzer)
- Bulk test data (/data/test-data)
All fake data tools (/data)

Was this tool useful?
YesNo

Canonical HTML: https://nutter.tools/data/ab-test-significance-calculator
Markdown version: https://nutter.tools/data/ab-test-significance-calculator.md
Plain-text version: https://nutter.tools/data/ab-test-significance-calculator.txt
Agent index: https://nutter.tools/llms.txt