all guides
A/B test sample size
A test you cannot finish is worse than no test: it ends on whichever side looked ahead that day. Enter your baseline and the lift you hope for, and see how many people it takes, how long that is at your traffic, and what a list your size can ever detect.
free toolthe test
what it takes
Rule of thumb (16 × p̄(1−p̄) / d²)
444 per arm
888 in total, about 5.9 weeks.
Exact formula
435 per arm
870 in total, about 5.8 weeks.
Fits in 5.8 weeks.
Under the 6 week line. Write the sample size down before you start, and do not read it early.
what your list can detect
How this works.
how this worksTwo ways to the same answer. The rule of thumb is quick enough to do on paper, with p̄ the average of the two rates and d the gap between them:
n per arm ≈ 16 × p̄(1 − p̄) / d² (16 for 80% power, 21 for 90%)
The exact formula uses the normal quantiles for a 5% two-sided test:
n = [ z(1−α/2) × √(2p̄(1−p̄)) + z(power) × √(p₀(1−p₀) + p₁(1−p₁)) ]² / d²
Weeks is both arms together divided by the units you can put through each week. If it comes to more than six weeks, do not run that test live: test a bigger change, pick a metric with a higher base rate, or replay old data instead of waiting. The last table runs the formula backwards: for 25, 50, 100, 200 or 400 people per arm, the smallest rate that arm size can tell apart from your baseline, found by bisection on the exact formula.
A small account can only detect big effects, so test changes big enough to double a rate and write the sample size down before you start. Both formulas assume one look at the end. Look every day and stop at the first win, and you will win by luck far more often than 5% of the time. The rule of thumb runs a little above the exact figure: 5% to 10% is 444 per arm by the rule and 435 exactly.
want this run for you? vibedraft finds the people ready to buy on X and in your own signups, drafts the right move in your voice, keeps your channels posting and learns what pays. you approve everything that goes out.
Start hereRuns in your browser. Nothing you type here is sent anywhere.