learn · experimentation engine

Assignment and randomization

Units, arms, holdouts and logged odds.

In Experimentation engine

Assignment decides which arm each unit gets: this lead gets the free brief, that one gets the waitlist link, this one gets nothing. Done by chance, sticky for the life of the test, and written down with the odds, it's what lets a difference between arms be read as cause instead of coincidence. On one small account it also has to deal with an audience that sees every arm at once.

registry → assignment · arm + odds → exposure log → metrics → readout → learning

rule 1 — Assign at eligibility, not at send. · A unit gets its arm the moment it qualifies. Assigning when a message is about to go out lets your own choices leak into who ends up where.

rule 2 — Same unit, same arm, every time. · A deterministic hash of the unit and the test's salt. No stored coin flips to lose, no unit seeing both arms.

rule 3 — Log the odds on every row. · The probability the unit had of landing in its arm. Without it, no new policy can ever be judged on old decisions.

what it is, in industry terms

Randomized assignment, sticky bucketing, logged propensities.

Product teams call this the assignment or bucketing service; statisticians call the result a randomized controlled design. The point of randomizing is that every other cause of the outcome (the lead's mood, the day's news, your own energy) ends up spread evenly across the arms, so only the knob differs on average.

Unit of randomization — The thing that gets an arm: a post, a lead, a visitor, an account, a day.

Unit of analysis — The thing the metric is counted on. It should match the unit of randomization, or the math needs correcting.

Bucketing — deterministic hashing · Hash the unit id with a per-test salt into a number from 0 to 9,999, then map ranges to arms.

Salt — A per-test string mixed into the hash, so two tests don't put the same units in the same arms.

Traffic allocation — Each arm's share. The holdout is a share too.

Stratification — blocking · Randomizing separately inside groups (tier, week, channel) so a small sample stays balanced on what matters.

Switchback — The whole account flips between arms by time period, in random order, for knobs that can't be split by unit.

Interference — SUTVA violation · One unit's arm changes another unit's outcome, as when both arms' posts reach the same followers.

Propensity — assignment probability · The odds of the arm a unit got, stored at decision time.

SRM — sample ratio mismatch · Arms that should be 50/50 come out 60/40. Almost always a bug, not luck.

how it works

From eligible unit to logged arm in five steps.

unit is eligible — lead enters Hot or Warm lead_7f3a
hash(salt : unit_id) — sha256 → first 8 hex → int mod 10,000
bucket — 6,412
0–999 — hold
control 45% — 1,000–5,499
variant 45% — 5,500–9,999 arm ranges, fixed at registration
unit row: arm=variant · p_assign=0.45 · bucket=6412 — assigned_at=eligibility time · salt=exp.campaign.offer.01

The unit becomes eligible
The population rule from the registration is checked the moment the unit appears: a lead enters Hot or Warm, a draft is generated, a visitor lands. Eligibility time is the assignment time.
Hash it with the test's salt
bucket = int(sha256(salt + ":" + unit_id)[0:8], 16) mod 10000. Same inputs, same bucket, forever. Each test has its own salt, so a lead in the variant of one test is not more likely to be in the variant of the next.
Map the bucket to an arm
Ranges are fixed at registration: 0 to 999 holdout, 1,000 to 5,499 control, 5,500 to 9,999 variant. Changing ranges mid-test reshuffles units, which is a new test.
Write the assignment row
Arm, p_assign (the arm's share, or the full probability vector for a bandit), bucket, stratum and block, salt, and the time. This happens even if the unit is never exposed.
Serve the arm when the moment comes
The sequence, draft or page looks up the arm instead of deciding again. Holdout units are blocked by the contact policy for the life of the test.

// one function, used by every surface that can expose a unit
import { createHash } from "node:crypto";

export function assign(spec, unitId) {
 const h = createHash("sha256").update(spec.salt + ":" + unitId).digest("hex");
 const bucket = parseInt(h.slice(0, 8), 16) % 10000;
 let lo = 0;
 for (const arm of spec.arms) { // [{name, share}], holdout first
 const hi = lo + Math.round(arm.share * 10000);
 if (bucket < hi) return { arm: arm.name, p_assign: arm.share, bucket };
 lo = hi;
 }
 throw new Error("arm shares must sum to 1");
}

the objects

The assignment spec on the test, and the assignment on each unit.

The spec is part of the registration and is frozen at gate A. Each assignment writes its result onto the unit's row in the exposure log; the fields below are the ones assignment owns.

Field	Type	What it holds
`salt`	string	Defaults to the experiment id. Never reused across tests.
`unit`	enum	post · lead · visitor · account · time slot · stored item · move.
`population`	rule	Who is eligible, evaluated at eligibility time with frozen inputs.
`arms`	[{name, share}]	Shares sum to 1, holdout included. Ranges derive from order and share.
`method`	enum	hash · stratified hash · block · switchback · cluster · stepped · bandit.
`strata`	list	For stratified designs: the fields to balance on, e.g. `tier`, `trigger`.
`block`	string	For block designs: the block definition, e.g. `3 days` or `ISO week`.
`assign_at`	enum	eligibility (default) or exposure_opportunity, with a reason.
`arm · p_assign · bucket`	on the unit row	The result: which arm, its odds, and the bucket that produced it.
`stratum · block_id`	on the unit row	Which stratum and block the unit fell in.
`assigned_at`	on the unit row	Eligibility time, not send time.

assignment spec, as data

{
 "experiment_id": "exp.campaign.offer.01",
 "salt": "exp.campaign.offer.01",
 "unit": "lead",
 "population": "tier in (Hot, Warm) and pain_signal_age_h < 72 and not never_contact",
 "method": "stratified hash",
 "strata": [
 "tier"
 ],
 "arms": [
 {
 "name": "holdout",
 "share": 0.1
 },
 {
 "name": "control",
 "share": 0.45
 },
 {
 "name": "brief",
 "share": 0.45
 }
 ],
 "assign_at": "eligibility"
}

one assignment, as data

{
 "unit_id": "lead_7f3a",
 "experiment_id": "exp.campaign.offer.01",
 "arm": "brief",
 "p_assign": 0.45,
 "bucket": 6412,
 "stratum": "tier=Hot",
 "block_id": null,
 "assigned_at": "2026-10-19T14:02:11Z",
 "exposed_at": null
}

choosing the unit

The unit decides what you can learn and what can go wrong.

Unit	Use it for	Interference risk	Count the metric on	Example
Post	openers, hooks, length, format, send time	high: one audience sees every post	the post (impressions at 24h, replies per post)
Lead	first move, offer, sequence length, follow-up timing	low to medium: leads rarely see each other's DMs, but public replies are visible	the lead (replied within 7 days, signed up within 14)	free brief vs waitlist link
Visitor	landing and waitlist copy, intake order	low: each visitor sees one page	the visitor (joined, finished intake)	pain headline vs product headline
Account	account-wide settings for many founders (later)	low between accounts, very high within one	the account (waitlist joins per week)	a new default cadence across many VibeDraft accounts
Time slot or day	knobs that can't be split by unit: posting cadence, the daily batch	carryover between periods	the slot or day, with a washout	two posts a day vs one, switched by day
Stored item	judge prompts, vocabularies, scorers (offline)	none: no one is exposed	the item (label agreement)	judge v2 replayed on 200 labeled matches
Move	which play the next-best-action engine picks	medium: moves on the same lead affect each other	the move, with the lead as a cluster	reply first vs DM first, chosen per move

contamination and interference on a small account

One audience sees every arm.

Textbook A/B tests assume each unit's outcome depends only on its own arm. On one founder's account that's often false: both arms' posts land in the same followers' feeds, the platform's ranking reacts to the account as a whole, and the founder gets better at writing week by week. These are the five ways it bites, and what to do about each.

Shared audience — Both arms' posts reach the same followers, so a strong variant post can lift the control post next to it, or tire the audience out. · block by time: each arm once per block, in random order, spaced apart; or switch by day for account-wide knobs.

Platform feedback — The ranking reacts to the account as a whole. A hot week lifts every arm, a cold week sinks every arm. · randomize inside each block, so good and bad weeks hit every arm equally; read results within blocks.

Founder drift — You get better at writing over the weeks, so whichever arm ran later looks better. · interleave arms across the whole period; never run arm A for two weeks and then arm B.

Spillover between leads — A control lead reads the public reply sent to a treatment lead in the same thread. · cluster by thread or community: everyone in one thread or subreddit gets the same arm.

Cross-channel leakage — A lead in the holdout sees your public post about the offer on another channel. · accept it and say so: holdout means no direct touch, not no exposure at all. Report public posting volume during the test.

the library

Ten assignment designs, and when to use each.

Pick the design at registration. Replay and shadow tests (see the sibling page) don't assign anything to people, so they aren't listed here.

design.ab-lead — lead

Simple randomized A/B

how: Hash each eligible lead into control or variant, 50/50.
odds: the arm's share
use: one act-side knob, enough leads per week
risk: none special; check SRM weekly
e.g.: reply first vs DM first on fresh pain signals

design.ab-holdout — lead

A/B with a holdout

how: Like above, plus a share (often 10%) that gets no touch at all.
odds: the share, holdout included
use: you need to know whether acting beats doing nothing
risk: a contact-policy gap lets someone touch a holdout lead
e.g.: free brief vs waitlist, with a 10% holdout

design.stratified — lead

Stratified by tier

how: Hash separately inside Hot, Warm and Watch so each tier splits evenly.
odds: share within the stratum
use: small samples where tier drives the outcome
risk: strata with very few units; merge them at registration

design.block-posts — post

Block-randomized posts

how: Fixed blocks of days; each arm runs once per block, in random order.
odds: 1 / number of arms, per block
use: post-level knobs on one account
risk: too few blocks to see anything; plan at least 3

design.slot — post

Time-slot randomization

how: Pick the slots first (say 8:04am and 12:04pm), then randomize which arm takes each slot.
odds: 0.5 per slot
use: format or headline tests where time of day matters
risk: slot effects swamp arm effects; balance slots across arms

design.switchback — time slot

Switchback by day

how: The whole account switches arm by day, in random order, with a washout day between.
odds: the share of days per arm
use: account-wide knobs: posting cadence, a new daily batch
risk: carryover from yesterday's arm; use washout days
e.g.: two posts a day vs one

design.cluster — community

Cluster by community

how: Randomize subreddits or threads, not people. Everyone in a cluster gets the same arm.
odds: the share of clusters
use: public replies that others in the thread can see
risk: few clusters means low power; count clusters, not people
e.g.: new reply style in half of the watched subreddits

design.stepped — lane or source

Stepped rollout

how: Turn a change on lane by lane in random order over weeks; each lane is its own before-and-after.
odds: order randomized at registration
use: detector or source changes you plan to ship everywhere anyway
risk: time trends; compare lanes still off with lanes already on

design.global-holdout — lead

Global holdout

how: A small share of all eligible leads is never touched by any campaign for a quarter.
odds: the global share
use: measuring the whole outbound program's lift, not one campaign's
risk: costs real signups; keep it small and time-boxed

design.bandit — post or lead

Bandit with logged odds

how: Thompson sampling shifts traffic toward the better arm as data arrives, logging the full probability vector.
odds: the full vector at each decision
use: after an A/B shows a real gap, to earn while learning
risk: biased estimates if odds aren't logged; never the first test

illustrative example

Assigning leads in the offer test, with a holdout.

The registered test exp.campaign.offer.01 compares a free brief with a waitlist link for Hot and Warm leads, 45% / 45%, with a 10% holdout that gets no touch.

A Reddit pain post is caught and judged real
The signal routes to the pain queue; the lead's tier becomes Hot. That moment is eligibility, so assignment runs now, before any draft exists.
Check the population
Tier Hot, pain signal 3 hours old, not on the never-contact list: eligible. The check uses the tier as it was at that moment, frozen.
Hash and map
sha256("exp.campaign.offer.01:lead_7f3a") gives bucket 6,412, inside the Hot stratum's brief range. Arm: brief. Odds: 0.45.
Write the row
Arm, odds, bucket, stratum tier=Hot, assigned at eligibility. The exposure time stays empty until the offer actually goes out.
Draft and send
The next-best-action engine asks for the lead's arm and drafts the brief offer. The founder sends it; the send carries a touch_id that ties the reply back to this unit.
Holdout leads
A lead that lands in 0 to 999 is marked holdout. The contact policy blocks every direct touch for the life of the test, and their signups are still counted: that's the baseline.

failure modes

How assignment goes wrong, and how to catch it.

Assigning at send time — Only leads you chose to message get an arm, so the arms differ in who you picked, not just in the offer. · every unit row's assigned_at should equal its eligibility time; the daily snapshot flags gaps.

Reassignment — A unit is drawn again later and lands in the other arm, so it sees both. · a deterministic hash makes this impossible; any stored random draw is a bug to remove.

Salt reuse — Two tests share a salt, so the same leads land in the variant of both and their effects get tangled. · the registry refuses a salt that has been used before.

Missing odds — Arms are recorded but not their probabilities, so off-policy estimates are impossible later. · p_assign is non-null on every unit row, enforced by the exposure log.

Sample ratio mismatch — Arms meant to be 50/50 come out 58/42: usually a filter applied after assignment, or an arm that crashes. · a chi-square test on arm counts at every snapshot; a p-value under 0.001 stops the readout until explained.

Leaky holdout — Someone messages a holdout lead by hand, or another campaign does. · the contact policy blocks holdout units for every campaign, and any touch on one is logged as a violation.

Imbalance on small samples — With 30 leads per arm, one arm gets most of the Hot leads by chance. · stratify on the fields that drive the outcome; report balance per stratum at readout.

data to capture

What assignment records, and what that trains.

Record — every assignment: unit, arm, odds (or the full vector), bucket, salt, stratum, block, eligibility time; every SRM check result; every holdout violation

Label — none on its own; assignments become labeled data once outcomes join through the exposure log

Trains — off-policy estimates for the playbook recommender (it needs the odds); uplift models (who is persuadable) from arm × outcome

Test — the assignment service itself: SRM rate across tests, share of unit rows with odds, holdout violation count (target: 0)