Arms and holdouts

in industry terms

Arms are variants; the holdout is the control that gets nothing.

This is a randomized controlled trial run inside a campaign. Marketing tools call the holdout a control group or a universal holdout when it spans every campaign. The assignment probability, the chance a lead had of landing in its arm, is called the propensity. Logging it is what lets later models be judged on old data.

how it works

From a frozen segment to logged assignments.

The assignment happens once per lead per campaign, before the first step runs. Hashing the lead id with the campaign id makes it repeatable: the same lead always lands in the same arm.

1. Frozen segment — member list
2. Stratify — tier, source
3. Hash — lead + campaign
4. Arm — treatment, holdout
5. Log — arm + probability
6. Steps — treatment only

arm = arms[ bucket( sha256(campaign_id + ":" + lead_id) ) ] # stable, no state to lose
bucket(h) = (h mod 10000) / 10000 → 0.00 to 0.89 treatment, 0.90 to 0.99 holdout
log: lead_assigned(campaign_id, lead_id, arm, probability = arm.share, stratum, method = "hash", at)

sizing for small lists

What a list of a few hundred can and can't tell you.

A common rule of thumb for comparing two rates (about 80% power, 5% false-positive rate) is n per arm ≈ 16 · p(1 − p) / d², where p is the average of the two rates and d the difference you want to see (the same convention as the readout page). The numbers below are worked examples of that formula, not real data.

the object

The arm, and the assignment.

Two records: the arm definition on the campaign, and one assignment row per lead.

one record, as data

{
 "campaign_id": "camp.lead-to-waitlist.2026-10-20",
 "arms": [
 {
 "name": "treatment",
 "share": 0.9,
 "changes": null
 },
 {
 "name": "holdout",
 "share": 0.1,
 "changes": "no messages for the window"
 }
 ],
 "assignment": {
 "lead_id": "lead_…",
 "arm": "holdout",
 "probability": 0.1,
 "method": "hash",
 "stratum": "source=reddit",
 "at": "2026-10-20T13:00:04Z"
 }
}

the catalog

Twelve arm designs, and when each fits.

Basic designs split people; cluster designs split threads or communities, because public replies are seen by everyone there; time designs split days or periods; program designs span every campaign.

illustrative example

Lead to waitlist, with a holdout.

unit

lead

Each lead gets private messages; public replies are a step, not the unit.

arms — treatment 90%, holdout 10%

stratify by — source (Reddit, X), so the holdout isn't all one platform

metric

waitlist joins per eligible lead, treatment vs holdout

Per eligible lead, not per offer: the holdout gets no offers.

what it can show — whether reaching out adds joins at all over people who'd find the waitlist anyway

failure modes

What goes wrong, and how to catch it.

How each one shows up, and the check that catches it before it costs a week.

data to capture

Record it so a model can learn from it later.

Follows the guide's eight data-spine rules: log the decision, version what decided, freeze the inputs, keep the negatives, log the odds, treat every human touch as a label, keep one chain of ids, keep the raw.

related pages

Where this connects.