all guides

Arms and holdouts

Variants and the group that gets nothing.

act · campaigns

Arms are the versions of a campaign that run side by side; the holdout is the arm that gets nothing. Together they turn a campaign from "we did a thing and some people joined" into "this caused that many joins". This page covers the randomization unit, assignment with logged odds, what a small list can and can't detect, and twelve arm designs.

campaign library → segment · who → offer · what they get → arms · who gets what → stop and readout · when it ends

Random, logged, sticky — Every assignment is random, written down with its probability, and never changed for that lead.

Pick the unit you measure — If replies happen per thread, randomize threads, not people.

Small lists read big effects only — Under a few hundred per arm, treat results as a direction unless the gap is large.

Arms are variants; the holdout is the control that gets nothing.

in industry terms

This is a randomized controlled trial run inside a campaign. Marketing tools call the holdout a control group or a universal holdout when it spans every campaign. The assignment probability, the chance a lead had of landing in its arm, is called the propensity. Logging it is what lets later models be judged on old data.

variant — Treatment arm · Gets the campaign: one offer, one sequence version.

variant — Second arm · The same campaign with one thing changed: the offer, the opener, the timing.

control — Holdout · Eligible, assigned at random, and left alone for the window. The baseline every arm is compared to.

From a frozen segment to logged assignments.

how it works

The assignment happens once per lead per campaign, before the first step runs. Hashing the lead id with the campaign id makes it repeatable: the same lead always lands in the same arm.

  1. Frozen segment — member list
  2. Stratify — tier, source
  3. Hash — lead + campaign
  4. Arm — treatment, holdout
  5. Log — arm + probability
  6. Steps — treatment only
arm = arms[ bucket( sha256(campaign_id + ":" + lead_id) ) ] # stable, no state to lose
bucket(h) = (h mod 10000) / 10000 → 0.00 to 0.89 treatment, 0.90 to 0.99 holdout
log: lead_assigned(campaign_id, lead_id, arm, probability = arm.share, stratum, method = "hash", at)

What a list of a few hundred can and can't tell you.

sizing for small lists

A common rule of thumb for comparing two rates (about 80% power, 5% false-positive rate) is n per arm ≈ 16 · p(1 − p) / d², where p is the average of the two rates and d the difference you want to see (the same convention as the readout page). The numbers below are worked examples of that formula, not real data.

Baseline rateDifference dLeads per armReads as
10%+10 points (to 20%)about 204a big effect, findable with a few hundred leads
5%+5 points (to 10%)about 444needs a month or more of Reddit leads
10%+3 points (to 13%)about 1,800out of reach for one founder; pool across campaigns or accept a direction
2%+2 points (to 4%)about 1,164typical cold reply rates are this low; test big changes only

Test changes big enough to double a rate. A 90/10 holdout on a few hundred leads gives a holdout of a few dozen: enough to see whether the campaign does anything, not enough to measure it precisely.

The arm, and the assignment.

the object

Two records: the arm definition on the campaign, and one assignment row per lead.

FieldTypeWhat it holds
arm.namestringtreatment, variant_b, holdout.
arm.share0 to 1The share of the segment it gets. Shares add to 1.
arm.changestextThe one thing that differs from the treatment arm.
assignment.lead_ididWho.
assignment.armstringWhere they landed. Never changed after assignment.
assignment.probability0 to 1The chance they had of landing there. For simple random splits, the arm's share.
assignment.methodenumhash · random_seeded · stratified · switchback · exploration
assignment.stratumtextThe stratum used, if any (tier, source).
assignment.attimestampWhen it happened, before the first step.

one record, as data

{
 "campaign_id": "camp.lead-to-waitlist.2026-10-20",
 "arms": [
 {
 "name": "treatment",
 "share": 0.9,
 "changes": null
 },
 {
 "name": "holdout",
 "share": 0.1,
 "changes": "no messages for the window"
 }
 ],
 "assignment": {
 "lead_id": "lead_…",
 "arm": "holdout",
 "probability": 0.1,
 "method": "hash",
 "stratum": "source=reddit",
 "at": "2026-10-20T13:00:04Z"
 }
}

Twelve arm designs, and when each fits.

the catalog

Basic designs split people; cluster designs split threads or communities, because public replies are seen by everyone there; time designs split days or periods; program designs span every campaign.

design.holdout-90-10 — Basic

Treatment and a 10% holdout

Everyone eligible gets the campaign except a random tenth.

unit
lead
answers
did the campaign cause anything at all
needs
a segment of at least a few hundred to read

design.ab-50-50 — Basic

Two variants, 50/50

Offer A vs offer B, or opener A vs opener B.

unit
lead
answers
which variant is better
needs
no holdout, so no absolute effect

design.ab-holdout — Basic

A/B plus holdout, 45/45/10

Both questions at once: does it work, and which version.

unit
lead
answers
lift of each arm over nothing
needs
the largest segments

design.thread-random — Cluster

Randomize by thread

Half the matching threads get an answer, half don't. Compare visits from each half.

unit
thread
answers
public reply lift
needs
threads, not people, are the unit of analysis

design.subreddit-cluster — Cluster

Randomize by community

Whole subreddits or Discords in or out, because one public reply is seen by everyone there.

unit
community
answers
lift where people can see each other's replies
needs
many communities; few units means low power

design.switchback — Time

Switchback by day

Treatment on some days, control on others, assigned at random.

unit
day
answers
effects that can't be split by person (posting cadence)
needs
several weeks

design.stagger — Time

Staggered rollout

Turn it on for half the targets now and half in two weeks. Compare while only half are on.

unit
target (competitor page, segment)
answers
lift of always-on plays like comparison pages
needs
patience; ends with everyone treated

design.pre-post — Time

Before and after, with a control period

Audience-wide campaigns (a launch) can't hold anyone out. Compare with the weeks before.

unit
period
answers
a direction, not a cause
needs
a stable baseline; label results as weak

design.stratified — Basic

Stratified random

Randomize separately inside each tier or source so arms stay balanced on small lists.

unit
lead within stratum
answers
same as A/B, with less luck in small samples
needs
the stratum stored with the assignment

design.global-holdout — Program

Global holdout, 5%

A fixed random slice of all leads that no campaign ever touches.

unit
lead, for months
answers
what the whole program adds over doing nothing
needs
discipline; never borrow from it

design.exploration — Program

Exploration slice

A small share of moves chosen at random among allowed options, with the odds logged.

unit
move
answers
feeds off-policy evaluation of the next-best-action engine
needs
propensity logging

design.matched-pairs — Cluster

Matched pairs

Pair leads that look alike, flip a coin inside each pair.

unit
pair
answers
sharper reads on very small lists
needs
a good matching rule (tier, source, age)

Lead to waitlist, with a holdout.

illustrative example

unit

lead

Each lead gets private messages; public replies are a step, not the unit.

arms — treatment 90%, holdout 10%

stratify by — source (Reddit, X), so the holdout isn't all one platform

metric

waitlist joins per eligible lead, treatment vs holdout

Per eligible lead, not per offer: the holdout gets no offers.

what it can show — whether reaching out adds joins at all over people who'd find the waitlist anyway

What goes wrong, and how to catch it.

failure modes

How each one shows up, and the check that catches it before it costs a week.

FailureHow it shows upHow to catch it
Unbalanced arms by luckHoldout ends up mostly Watch-tier or one sourceStratify, and check balance on tier and source right after assigning
ContaminationHeld-out people see your public reply in the same threadRandomize by thread or community for public steps
Unit mismatchRandomized by lead, measured by threadWrite the unit on the experiment and measure in the same unit
ReassignmentA lead moves arms after a rule changeHash-based assignment and a rule that assignments are immutable
PeekingCalling the winner on day 3Readout date fixed at launch; only guardrails are checked early
Borrowing from the holdout"Just this once" messaging held-out Hot leadsHoldout leads excluded by the contact policy for the window

Record it so a model can learn from it later.

data to capture

Follows the guide's eight data-spine rules: log the decision, version what decided, freeze the inputs, keep the negatives, log the odds, treat every human touch as a label, keep one chain of ids, keep the raw.

record

  • every assignment: campaign, lead, arm, probability, method, stratum, time
  • balance checks right after assignment (tier, source, signal age by arm)
  • holdout leads' outcomes, collected the same way as treated leads'
  • any lead whose arm was overridden, and why

label

conversion per arm; lift over the holdout

trains

uplift models (who is persuadable) and off-policy evaluation of new targeting rules

test

arm designs themselves: holdout share, stratification, unit · randomized assignment with logged propensities

More in Campaigns