Readout and statistics

what it is, in industry terms

Statistical inference with a decision rule attached.

Experimentation platforms call this the analysis or readout step. It answers three questions in order. Did the test run the way it was designed (enough units, the right split)? Is the difference real or noise? Is it big enough to act on? Every number it uses comes from the experiment record, which was written before launch.

how it works

Seven checks, always in the same order.

The readout runs on the judge-by date whether the test looks interesting or not. A failed early check stops the readout before the primary metric is ever looked at, so a broken test can't be read as a win.

1. Freeze — the snapshot on the judge-by date, never later
2. Sample — reached? units per arm vs min_per_arm
3. Split — as planned? SRM check: χ² on assigned counts
4. Exposure — clean? assigned but never shown is excluded
5. Guardrails — any breach stops the readout
6. Primary — vs the rule effect + interval, method set at launch
7. Verdict — + learning ship · revert · extend · kill · void A failed check at steps 2 to 4 ends the readout as void or underpowered, before the primary metric is computed.

1. Freeze — Copy every unit, its arm, its exposure time and its outcomes into a readout snapshot at the judge-by date. Late events after that date don't count.

2. Sample reached? — Compare units per arm to min_per_arm. If short, the primary metric isn't read: extend once, as pre-planned, or close as underpowered.

3. Split as planned? — A chi-square test on assigned counts against the planned shares. p < 0.001 means sample ratio mismatch, usually a bug in assignment or logging. The test is void.

4. Exposure clean? — Only units that actually saw their arm count. A lead assigned a DM that was never sent is excluded from both the count and the outcome.

5. Guardrails — Stop requests, spam reports, founder minutes, tool spend. A breach ends the test as revert, whatever the primary metric says.

6. Primary vs the rule — Compute the effect and its interval with the method named in the record, then apply the win rule exactly as written.

7. Verdict and learning — Write the verdict, the effect with its interval, and the confidence label. Then write one learnings row, win or lose.

sizing for small accounts

How many units each arm needs, before launch.

For a yes or no metric, the rule of thumb for 80% power at a 5% two-sided significance level is n per arm ≈ 16 · p(1 − p) / d², where p is the average rate across the two arms and d is the difference you want to detect. The exact two-proportion formula is below; the rule of thumb approximates it.

n ≈ 16 · p̄(1 − p̄) / d² rule of thumb (80% power, α = 0.05) n = [ z₁₋α/₂ · √(2p̄(1 − p̄)) + z₁₋β · √(p₀(1 − p₀) + p₁(1 − p₁)) ]² / d² exact, two proportions n ≈ 16 · σ² / d² for an average (use log impressions, not raw)

why peeking breaks a test

Look every day and stop at the first "win", and you'll win by luck far more than 5% of the time.

A fixed-horizon test promises a 5% false-positive rate only if you look once, at the end. Each extra look is another chance for noise to cross the line, and you stop on the lucky one. A run of 2,000 A/A tests, where both arms are identical so every "win" is false, shows how often each habit declares a winner.

Ways to look safely

• Don't look at the primary metric. Watch guardrails only (stop requests, spam reports, credits). Stopping for harm is always allowed.
• Plan the looks. A group-sequential design (O'Brien-Fleming boundaries) spends the 5% across, say, 3 planned looks, with a very strict line early and close to the normal line at the end.
• Use an always-valid test. A mixture sequential probability ratio test (mSPRT) gives p-values you can check any time, at the cost of needing a bit more data.
• Read a Bayesian posterior without a stop rule. Report it on the judge-by date only. Bayesian numbers don't make peeking free if you still stop on the first good-looking day.

A sensible default

Fixed horizon, block-randomized, one readout on the judge-by date. Guardrails are checked daily and can stop a test for harm. The primary metric isn't computed until judge-by. That's the cheapest rule to follow and the easiest to audit.

Bayesian readouts

"How likely is the variant better, and by how much?" with priors from past tests.

For a yes or no metric, each arm's rate gets a Beta prior. Data updates it: successes add to α and failures add to β. The posterior answers the founder's real question directly: the chance the variant is better, a range for the difference, and the expected loss if you ship it and you're wrong. Small samples give wide answers, honestly.

prior: rate ~ Beta(α₀, β₀) α₀ = m · s, β₀ = (1 − m) · s (m = prior mean, s = prior strength in pseudo-units) posterior: rate ~ Beta(α₀ + successes, β₀ + failures) report: P(variant > control), 90% interval for (variant − control), expected loss = E[max(control − variant, 0)]

pre-set win rules

A win rule is a test too. Know its false-win rate.

Hand-run tests on a small account often use simple rules instead of p-values: "both posts clear 150 impressions at 24 hours and the other version's don't", or "beats both other arms in 2 of 3 blocks". Those rules are good: they're written in advance and easy to read. But each one has a chance of declaring a winner when nothing is different, and that chance should be known before launch. The table works it out for the rules in use, assuming the units are independent and every arm truly performs the same.

the decision table

Every readout ends in one of six verdicts.

The verdict depends on three things, read in order: did the test run cleanly, did a guardrail break, and where does the interval for the primary metric sit compared with the smallest effect worth acting on? That smallest effect is set before launch, not after.

the readout object

One frozen snapshot per readout.

The readout is written once, at the judge-by date, and never edited. Recomputing it later from live tables would let late events and fixed bugs quietly change history. A re-analysis is a new readout row that points at the old one.

{
  "readout_id": "ro.exp.post.opener.01.1",
  "experiment_id": "exp.post.opener.01",
  "frozen_at": "2026-05-12T21:00:00Z",
  "n_assigned": {
    "proof": 2,
    "plain": 2
  },
  "n_exposed": {
    "proof": 2,
    "plain": 2
  },
  "srm_p": null,
  "guardrails": [
    {
      "metric": "negative_feedback",
      "proof": 0,
      "plain": 0,
      "limit": 0,
      "broke": false
    }
  ],
  "primary": {
    "metric": "impressions_24h >= 150",
    "value": {
      "proof": "2 of 2",
      "plain": "0 of 2"
    },
    "raw": {
      "proof": [
        310,
        188
      ],
      "plain": [
        74,
        51
      ]
    },
    "method": "threshold rule, set before launch"
  },
  "secondary": {
    "replies": {
      "proof": 14,
      "plain": 3
    }
  },
  "rule": {
    "text": "both posts in a version clear 150 impressions at 24h and the other version's don't",
    "result": "pass: proof"
  },
  "looks": 1,
  "verdict": "ship",
  "confidence": "suggestive",
  "caveats": [
    "metrics pulled between 20 and 30 hours after posting, not exactly 24h",
    "other posts went out the same week and shared the audience",
    "the post bodies differed, so the opener isn't perfectly isolated",
    "rule's false-win rate under no difference is at most 12.5%"
  ],
  "supersedes": null
}

Illustrative values, invented to show the shape.

catalog of methods

Which method for which metric.

Pick one before launch and write it into the experiment record. Small-account fit says how well each one copes with a few dozen units.

failure modes

How readouts go wrong, and how to catch it.

data to capture

What the readout records, and what that trains.

related pages

Where the inputs come from and where the verdict goes.

Formulas are standard approximations; the worked examples assume independent units. The 7/9 and 12.5% figures assume every arm truly performs the same and ties are broken at random.