all guides

A/B test sample size

A test you cannot finish is worse than no test: it ends on whichever side looked ahead that day. Enter your baseline and the lift you hope for, and see how many people it takes, how long that is at your traffic, and what a list your size can ever detect.

free tool

the test

What the thing converts at today.
Test changes big enough to double a rate.
Power
People or messages you can put through the test each week, both arms together.

what it takes

Rule of thumb (16 × p̄(1−p̄) / d²)

444 per arm

888 in total, about 5.9 weeks.

Exact formula

435 per arm

870 in total, about 5.8 weeks.

Fits in 5.8 weeks.

Under the 6 week line. Write the sample size down before you start, and do not read it early.

what your list can detect

25Weeks to collect0.3 weeksSmallest rate it can see36.3%Lift on baseline+627%
50Weeks to collect0.7 weeksSmallest rate it can see24.7%Lift on baseline+394%
100Weeks to collect1.3 weeksSmallest rate it can see17.4%Lift on baseline+249%
200Weeks to collect2.7 weeksSmallest rate it can see13%Lift on baseline+160%
400Weeks to collect5.3 weeksSmallest rate it can see10.2%Lift on baseline+105%

How this works.

how this works

Two ways to the same answer. The rule of thumb is quick enough to do on paper, with p̄ the average of the two rates and d the gap between them:

n per arm ≈ 16 × p̄(1 − p̄) / d²
(16 for 80% power, 21 for 90%)

The exact formula uses the normal quantiles for a 5% two-sided test:

n = [ z(1−α/2) × √(2p̄(1−p̄)) + z(power) × √(p₀(1−p₀) + p₁(1−p₁)) ]² / d²

Weeks is both arms together divided by the units you can put through each week. If it comes to more than six weeks, do not run that test live: test a bigger change, pick a metric with a higher base rate, or replay old data instead of waiting. The last table runs the formula backwards: for 25, 50, 100, 200 or 400 people per arm, the smallest rate that arm size can tell apart from your baseline, found by bisection on the exact formula.

A small account can only detect big effects, so test changes big enough to double a rate and write the sample size down before you start. Both formulas assume one look at the end. Look every day and stop at the first win, and you will win by luck far more often than 5% of the time. The rule of thumb runs a little above the exact figure: 5% to 10% is 444 per arm by the rule and 435 exactly.

want this run for you? vibedraft finds the people ready to buy on X and in your own signups, drafts the right move in your voice, keeps your channels posting and learns what pays. you approve everything that goes out.

Start here

Runs in your browser. Nothing you type here is sent anywhere.