learn · experimentation engine

Exposure log and units

Which unit saw which arm, joined to outcomes.

In Experimentation engine

The exposure log has one row per unit in a test: the post, lead, visitor or stored item, the arm it was given, the odds, when it actually saw its arm, and the ids that join it to what happened next. A readout is a query over these rows. If a unit is missing here, it's missing from the result, and that's usually how tests quietly lie.

registry → assignment → exposure log · unit rows → metrics → readout → learning

rule 1 — Every assigned unit gets a row. · Including the ones never exposed and the holdout. Dropping them changes who is being compared.

rule 2 — Analyze as assigned. · Intention-to-treat is the default verdict. Exposed-only numbers are reported beside it, never instead of it.

rule 3 — Join by id, not by time. · Outcomes attach through the touch_id, post_id or lead_id, never "the reply that came in that week".

what it is, in industry terms

Assignment, exposure and outcome, kept apart.

Experimentation platforms separate three moments: when a unit is assigned an arm, when it's exposed to it, and when its outcome is observed. Most test bugs come from blurring them, by counting only the units that got exposed, or by joining outcomes on a time window that picks up other campaigns' results.

Exposure — The moment the unit actually met its arm: the post went live, the DM was sent, the page rendered.

Intention-to-treat — ITT · Compare arms as assigned, exposed or not. Unbiased, slightly diluted.

Triggered analysis — per-protocol · Compare only exposed units. Sharper, but biased unless exposure has nothing to do with the arm.

Counterfactual logging — For control units, log the moment they would have been exposed, so triggered analysis compares like with like.

Outcome window — How long after exposure an outcome still counts: 24 hours for a post's reach, 7 days for a reply, 14 for a signup.

Attrition — Units whose outcome can't be observed: a deleted post, a blocked DM, a disconnected account.

Dilution — Unexposed units in ITT shrink the measured effect toward zero. Honest, but plan for it in sizing.

Join key — The id that ties a unit to its outcomes: touch_id for messages, post_id for posts, visitor id for pages.

how it works

One unit's life, from arm to counted outcome.

Assignment writes the row
At eligibility: experiment id and version, unit id and type, arm, odds, bucket, stratum, block. exposed_at is empty.
Exposure fills in the moment and the join id
When the unit meets its arm (the DM is sent, the post goes live, the page renders), the exposure event writes exposed_at, the exposure type and the id: touch_id, post_id or page view id.
Control gets a counterfactual exposure
When a control unit reaches the point where the variant would have been shown, log would_expose_at. That's what lets an exposed-only diagnostic compare like with like.
Outcomes arrive through the ids
Replies, clicks, joins and payments carry the same ids, so they attach to the unit without guessing. Only outcomes inside the registered window count.
The window closes, the readout snapshots
At judge-by, units whose windows have closed get their outcome snapshot frozen. Units still inside their window are listed as pending, never counted as zeros.

the object

The experiment unit, field by field.

This is the full row behind the short version on the parent page. Assignment writes the first half, exposure writes exposed_at and the join ids, and the readout fills the outcome snapshot once the window closes.

Field	Type	What it holds
`unit_id · unit_type`	string · enum	The post id, lead id, visitor id or stored item id, and which kind.
`experiment_id · experiment_version`	string · hash	The test, and the `registered_hash` it was assigned under.
`arm · p_assign · bucket`	string · 0 to 1 · int	From assignment. Odds are never null.
`stratum · block_id`	string	From assignment, for stratified and block designs.
`eligible_at · assigned_at`	timestamp	Equal by default. A gap is a bug unless the registration says why.
`exposed_at · exposure_type`	timestamp · enum	When and how the unit met its arm. Null if it never did.
`would_expose_at`	timestamp	Control only: when it would have been exposed. Enables a fair exposed-only diagnostic.
`join_ids`	object	`touch_id`, `post_id`, `lead_id`, `signal_id`, `visitor_id`, whichever apply.
`window_end`	timestamp	exposed_at (or assigned_at when unexposed) plus the registered outcome window.
`outcome`	object	Frozen at readout: the primary metric and each guardrail for this unit, plus the event ids behind them.
`attrition`	code or null	deleted · undeliverable · blocked · disconnected · window_open.
`excluded · exclusion_reason`	bool · code	Only reasons named in the registration's analysis plan.
`notes`	text	Anything a human did by hand, such as a founder override, recorded rather than hidden.

a lead unit, as data

{
 "unit_id": "lead_7f3a",
 "unit_type": "lead",
 "experiment_id": "exp.campaign.offer.01",
 "experiment_version": "rh_9c1e",
 "arm": "brief",
 "p_assign": 0.45,
 "bucket": 6412,
 "stratum": "tier=Hot",
 "eligible_at": "2026-10-19T14:02:11Z",
 "assigned_at": "2026-10-19T14:02:11Z",
 "exposed_at": "2026-10-19T16:40:05Z",
 "exposure_type": "dm_sent",
 "join_ids": {
 "lead_id": "lead_7f3a",
 "touch_id": "t_51d0",
 "signal_id": "sig_88b2"
 },
 "window_end": "2026-11-02T16:40:05Z",
 "outcome": {
 "signed_up_14d": true,
 "replied_7d": true,
 "events": [
 "reply_received:ev_31a",
 "signup_created:ev_77f"
 ]
 },
 "attrition": null,
 "excluded": false
}

a post unit, as data

{
 "unit_id": "post_9f1a",
 "unit_type": "post",
 "experiment_id": "exp.sequence.opener.01",
 "arm": "proof",
 "p_assign": 0.5,
 "block_id": "week 1",
 "assigned_at": "2026-10-19T12:04:00Z",
 "exposed_at": "2026-10-19T12:04:31Z",
 "exposure_type": "post_published",
 "join_ids": {
 "post_id": "post_9f1a",
 "draft_id": "d_4410"
 },
 "window_end": "2026-10-20T12:04:31Z",
 "outcome": {
 "impressions_24h": null,
 "cleared_140": null,
 "replies_24h": null,
 },
 "attrition": null
}

exposure vs assignment

Three ways to read the same rows.

The verdict uses intention-to-treat. The other two are diagnostics: they explain a result, they don't decide it.

Analysis	Who is counted	Biased when	Use it to
Intention-to-treat (the verdict)	every assigned unit, by assigned arm	never, if every unit has a row	decide. This is what the win rule reads.
Exposed-only	units with exposed_at set	exposure depends on the arm (a heavier offer gets sent less)	explain a diluted result; never to decide
Triggered, with counterfactual	exposed treatment units and control units with would_expose_at	the would-expose point is logged differently from real exposure	get a sharper estimate when exposure rates are low, if logging is symmetric

why exposed-only lies

Say the brief arm takes the founder 20 minutes per lead, so on busy days only the easiest Hot leads get their brief sent. Counting only exposed units compares the easiest Hot leads with everyone in control. The brief "wins" even if it does nothing. Intention-to-treat keeps the unsent brief leads in the denominator, which is exactly the cost of choosing a heavier offer.

joining units to outcomes

The id chain, from unit to payment.

Each outcome attaches through one id. Where the id doesn't exist yet, the fallback is a time-window join on the lead, which is weaker: if two campaigns touch the same lead, a reply can't be credited to either with certainty.

Outcome	Joins on	Fallback today
Reply to a DM or public reply	touch_id on the send, carried on the reply	lead_id plus a time window after the send
Click on a link	the short link's id, minted per touch
Waitlist join or signup	first and last touch_id on the signup	matching the email or handle to a lead
Payment	Stripe customer → account → first touch	Stripe webhooks exist; the touch half doesn't
Post reach		none needed
Draft decision	draft_id → decision, edit distance, edit locus, seconds to decide	none needed
Judge or detector verdict (replay)	stored item id → founder label	none needed once labels are stored

the library

Twelve kinds of exposure, and what proves each one.

An exposure is only real if an event proves it. Each row names the event, the id it writes, and the default outcome window.

exposure.post — post

Post published

proof: post.published
writes: post_id (tweet id)
joins: impressions, replies, profile clicks, bookmarks
window: 24 hours (snapshots out to 7 days)

exposure.thread — post

Thread or chain published

proof: post.published on the root
writes: root post_id
joins: the root's metrics; replies anywhere in the chain
window: 24 hours

exposure.quote — post

Quote or repost from a second account

proof: post.published on the quoting account
writes: quote post_id, linked to the target post
joins: engagements on the target post
window: 24 hours

exposure.public-reply — message

Public reply to a lead

proof: reply posted
writes: touch_id + reply post_id
joins: their reply, profile click, follow
window: 7 days

exposure.dm — message

DM sent

proof: founder confirms the send
writes: touch_id
joins: reply, link click, join
window: 7 days

exposure.email — message

Email delivered

proof: delivery webhook
writes: touch_id
joins: open (weak), click, reply
window: 7 days

exposure.keyword — message

Comment-a-keyword auto-reply

proof: auto-reply sent
writes: touch_id
joins: click, join
window: 7 days

exposure.brief — offer

Free brief delivered

proof: brief link created and sent
writes: touch_id + brief id
joins: brief viewed, join
window: 14 days

exposure.page — visitor

Page or variant rendered

proof: page view with the variant id
writes: visitor id + view id
joins: waitlist join, intake finished
window: the session, up to 7 days

exposure.intake — visitor

Intake step shown

proof: intake step reached
writes: visitor id
joins: next step reached, finished
window: the session

exposure.replay — offline

Stored item replayed

proof: replay run completes
writes: item id + version
joins: verdict vs founder label
window: none: label already exists

exposure.shadow — offline

Shadow verdict produced

proof: shadow run on a live item
writes: item id + shadow version
joins: later founder label on the same item
window: until labeled

attrition

When the outcome can't be seen.

Attrition is fine when it hits every arm the same way. It's a problem when one arm causes more of it, like a pushier message that gets more blocks. Every attrition reason is a pre-named code, counted per arm at readout.

Reason	Example	Under ITT	Flag when
deleted	the post was deleted before its 24-hour snapshot	keep the unit; outcome unknown, reported per arm	more deletions in one arm
undeliverable	the DM couldn't be sent: closed DMs, a suspended account	keep it; an outcome of no reply is the honest ITT value	delivery fails more in one arm
blocked	the lead blocked or reported you	keep it, and count it as a guardrail hit too	any block in a variant arm gets a look
disconnected	your account lost its sign-in mid-test, so snapshots stopped	keep it; mark the outcome unknown; extend judge-by if the registration allows	more than 5% of units in any arm
window_open	judge-by came before the outcome window closed	list as pending; never count as zero	pending units in one arm more than the other

failure modes

How the exposure log goes wrong, and how to catch it.

Dropping unexposed units — The table only gets a row when something is sent, so units that never got their arm vanish. · rows are created at assignment; a daily check compares assigned counts with the registry's expected flow.

Counting impressions as units — A post with 1,000 impressions is treated like 1,000 data points. · the unit type is fixed at registration; metrics aggregate to the unit first, then compare arms.

Time-window joins — A reply in the week after a DM is credited to it, even if another campaign also touched that lead. · join on touch_id; until it exists, label every lead-level readout "joined by time window".

Counting open windows as zero — Judge-by arrives three days after the last send, and 7-day reply windows are counted as no reply. · units with window_end after the snapshot are pending, never zero.

data to capture

What the exposure log records, and what that trains.

Record — one row per assigned unit, holdout and unexposed included; exposure and counterfactual-exposure times; join ids; frozen outcome snapshots; attrition codes

Label — each unit's outcome inside its window, joined by id: the training label for uplift and response models

Trains — uplift models (who responds to which arm); reply propensity using the arm as a feature; off-policy estimates with p_assign

Test — the log itself: share of units with join ids, share pending at readout, SRM between assigned and exposed counts per arm