learn · experimentation engine

Metrics and guardrails

Primary metrics and what must not get worse.

In Experimentation engine

A test is only as good as the number that decides it. Every metric is defined once, in one place, with its formula, its unit, its window and the events it reads. Each test picks exactly one primary metric to decide on, and a few guardrails that must not get worse: stop requests, blocks, founder minutes, tool spend. Everything else is context.

registry → assignment → exposure log → metrics · defined once → readout → learning

rule 1 — One primary per test. · The win rule reads one number. Secondary metrics are reported, never decisive.

rule 2 — Windows start at exposure. · Not on the calendar. A lead messaged on Friday gets the same seven days as one messaged on Monday.

rule 3 — Every guardrail has a kill line. · Written at registration. Crossing it stops the test, whatever the primary looks like.

what it is, in industry terms

A metric layer: definitions, not dashboards.

Experimentation teams keep a metric catalog (sometimes called a semantic layer) so that "reply rate" means the same thing in every test, every readout and every report. The vocabulary below is standard; what's specific here is that the units are small, so the metric choice decides whether a test can say anything at all.

Primary metric — OEC · overall evaluation criterion · The one number the win rule is written on.

Secondary metric — Reported beside the primary to explain it. Never changes the verdict.

Guardrail — guardrail metric · A number that must not get worse past a kill line: blocks, stop requests, cost, founder time.

Diagnostic — A health check on the test itself: sample ratio, attrition, exposure rate.

Window — How long after exposure an outcome counts. Part of the metric's definition, not a setting at readout.

Threshold metric — A yes or no per unit: "cleared 150 impressions at 24 hours". Turns a noisy count into something a small sample can compare.

Ratio metric — A sum over a sum, like replies per touch. Needs care when the unit you randomized is not the unit in the denominator.

Proxy — surrogate metric · A faster, more frequent number standing in for a slow one: replies for signups.

Novelty effect — A change gets attention because it's new, then fades. Early weeks overstate it.

Goodhart's law — When a measure becomes a target, it stops being a good measure. Guardrails are the defense.

how it works

From events to a verdict, in four passes.

events — reply_received post snapshots signup_created
per-unit value — inside the unit's window replied_7d = 1
per-arm aggregate — mean over all assigned control 0.05 · brief 0.14
difference — with a 95% interval +0.09 [0.02, 0.16]
verdict checks — ✓ primary vs win rule ✓ every guardrail under its kill line ✓ sample ratio ok ✓ units per arm met illustrative numbers · the unit of analysis is the unit that was randomized

Events in
Anything without an id is a diagnostic, not a metric.
Per-unit value, inside the window
Each unit gets one value per metric: 1 or 0 for threshold and yes-or-no metrics, a count or amount otherwise, counted only inside its window from exposure (or assignment, for unexposed units).
Per-arm aggregate over every assigned unit
The mean across all assigned units, exposed or not (intention-to-treat). Pending units are listed, not averaged in.
Difference, interval, checks
The difference between arms with a 95% interval, then the verdict checks: the win rule on the primary, every guardrail against its kill line, the sample ratio, units per arm. Any failed check blocks a "win".

the object

A metric definition, field by field.

Tests refer to metrics by id. Changing a definition creates a new version; tests already registered keep the version they were registered with.

Field	Type	What it holds
`metric_id · version`	string · int	`replied_7d` v1. Tests store the version they were registered with.
`name · question`	text	"Replied within 7 days" · "Did the lead answer?"
`role`	primary-capable · guardrail · diagnostic	What it's allowed to do in a test.
`unit`	post · lead · visitor · draft · stored item · day	What it's counted on. Must match the unit of randomization or say how it's corrected.
`formula`	expression	Per unit first, then aggregate: `any(reply_received where touch_id in unit.touches)`.
`window`	duration from exposure	7 days, 24 hours, the session.
`direction`	up · down	Which way is better.
`source_events`	event types + join key	`reply_received` joined on `touch_id`.
`kind`	binary · count · amount · ratio · threshold	Decides the variance formula at readout.
`kill_default`	rule	Guardrails only: the default kill line a registration starts from.
`proxy_for`	metric id	If it stands in for a slower metric, which one, and the proxy-check status.
`owner · defined_at`	person · date	Who defined it, and when.

one metric, as data

{
 "metric_id": "replied_7d",
 "version": 1,
 "name": "Replied within 7 days",
 "role": "primary-capable",
 "unit": "lead",
 "kind": "binary",
 "formula": "1 if any reply_received joined on a touch_id of this unit within window, else 0",
 "window": "7d from exposed_at (assigned_at if unexposed)",
 "direction": "up",
 "source_events": [
 {
 "type": "reply_received",
 "join": "touch_id"
 }
 ],
 "proxy_for": "signed_up_14d",
 "proxy_check": "unvalidated",
 "defined_at": "YYYY-MM-DD"
}

one guardrail, as data

{
 "metric_id": "stop_requests_per_100_touches",
 "version": 1,
 "role": "guardrail",
 "unit": "lead (touches counted per arm)",
 "kind": "ratio",
 "formula": "100 * stop-class replies / touches sent, per arm",
 "window": "life of the test",
 "direction": "down",
 "source_events": [
 {
 "type": "reply_received",
 "filter": "class = stop",
 "join": "touch_id"
 },
 {
 "type": "touch_sent",
 "join": "touch_id"
 }
 ],
 "kill_default": "> 2 per 100 touches in any arm, with at least 50 touches → stop"
}

the four roles

Role	How many per test	Decides the verdict?	Rule
Primary	exactly 1	yes, through the win rule	defined before launch; its window and unit match the registration
Secondary	0 to 3	no	explains the primary; a big secondary move becomes the next test's hypothesis
Guardrail	1 or more	only as a veto	each has a kill line; crossing it stops the test and blocks a win
Diagnostic	always on	no, but can invalidate	sample ratio, exposure rate, attrition, pending units; a failure means fix and rerun

the three traps, in depth

Windows, ratios and novelty.

Windows start at exposure

"Replies this week" gives a lead messaged Monday seven days and a lead messaged Friday three. If one arm's sends cluster late in the week, it loses for no reason. Each metric carries its window, measured from the unit's own exposure, and units whose window hasn't closed are pending at readout, never zero.

Unit	Default window	Why
post	24 hours	reach mostly settles in a day; snapshots run out to 7 days
lead (reply)	7 days	late replies to a cold message are rare
lead (signup)	14 days	a decision takes longer than a reply
visitor	the session, up to 7 days	joins happen on the visit or a return visit
draft	until decided	the decision is the outcome

Ratio metrics need the right unit

If you randomize leads but count "replies per touch", a lead who got four messages counts four times, and touches from one lead aren't independent. The interval comes out too narrow and false wins follow. Two fixes: switch to a per-unit metric (replied_7d, yes or no per lead), which is the default, or keep the ratio and compute its variance with the delta method.

ratio R = mean(Y) / mean(X), per arm, over n randomized units
 Y = replies per lead, X = touches per lead

Var(R) ≈ (1/n) · [ Var(Y)/μX² − 2·μY·Cov(X,Y)/μX³ + μY²·Var(X)/μX⁴ ]

compare arms with (R_b − R_a) ± 1.96 · sqrt(Var(R_a) + Var(R_b))

Novelty fades

A new opener style or a new page gets a lift partly because it's new to the people who see it often. On one account, your followers see every post, so novelty is real. Read the primary for the first and second half of the test separately; if the second half's effect is less than half the first's, label the result "novelty suspected" and extend if the registration allows.

Proxies need a check

Signups are rare, so tests often decide on replies. That's fine if replies predict signups. Record both in every lead test. Across tests, compare the reply lift with the signup lift; until enough tests agree, every decision made on a proxy is labeled "decided on proxy" in the learnings base.

the library

Sixteen metrics, each defined once.

Ten that can decide a test and six guardrails. Kill lines are worked defaults to set per test at registration, not measured thresholds.

replied_7d — primary

Replied within 7 days

unit: lead
formula: 1 if any reply joined on the unit's touch_ids
window: 7 days from exposure · higher is better
events: reply_received · touch_id
stage: 8 to 10

signed_up_14d — primary

Signed up within 14 days

unit: lead
formula: 1 if a waitlist join or signup carries one of the unit's touch_ids
window: 14 days · higher is better
events: signup_created · first and last touch
stage: 9

cleared_150_24h — primary

Cleared 150 impressions at 24h

unit: post
formula: 1 if impressions at the 1,440-minute snapshot are 150 or more
window: 24 hours · higher is better
stage: 10

engagements_24h — primary

Engagements at 24h

unit: post
formula: likes + replies + bookmarks + profile visits + follows gained, excluding your own account's actions
window: 24 hours · higher is better
stage: 10

resonance_100 — secondary

Resonance per 100 impressions

unit: post group
formula: 100 × (replies + bookmarks + profile visits) / impressions, pooled over the group's posts
window: 24 hours · higher is better
stage: 10

waitlist_join_rate — primary

Waitlist join rate

unit: visitor
formula: 1 if the visitor joins the waitlist
window: the session, up to 7 days · higher is better
events: site visit · waitlist join
stage: 11

finished_intake — primary

Finished the intake

unit: visitor
formula: 1 if the visitor reaches the dashboard after the brief
window: the session · higher is better
events: intake step events
stage: 1

fields_kept — primary

Brief fields kept unedited

unit: brief field
formula: 1 if the founder keeps the field as extracted
window: until the brief is confirmed · higher is better
events: brief statuses · founder claims
stage: 2

precision_top20 — primary

Precision in the daily top 20

unit: stored match
formula: share of the day's top 20 matches the founder labels real
window: per day · higher is better
events: founder match labels
stage: 5 to 7

shipped_unedited — primary

Drafts shipped unedited

unit: draft
formula: 1 if decision is shipped, not shipped edited or discarded
window: until decided · higher is better
events: drafts.decision
stage: 10

stop_requests — guardrail

Stop requests per 100 touches

unit: lead
formula: 100 × stop-class replies / touches sent, per arm (ratio: delta method)
window: life of the test · lower is better
events: reply class · touch_sent
kill: > 2 per 100 with 50+ touches
stage: 8 to 10

negative_feedback — guardrail

Blocks, reports and mutes

unit: lead or post
formula: count per arm
window: life of the test · lower is better
events: platform signals where available
kill: any block in a variant arm triggers a review
stage: 8 to 10

founder_minutes — guardrail

Founder minutes per day

unit: day
formula: minutes spent on the test's units: draft decisions plus logged manual time
window: per day · lower is better
events: drafts.time_to_decide_s · manual log
kill: > 30 a day for 3 days
stage: all

credits_per_unit — guardrail

Credits per unit

unit: unit
formula: tool spend for the unit's drafts, previews and sends
window: life of the test · lower is better
events: credit ledger
kill: more than 2× control
stage: all

policy_violations — guardrail

Contact-policy violations

unit: lead
formula: touches that broke a cap, quiet hours, never-contact or a holdout
window: life of the test · lower is better
events: contact policy checks · touch_sent
kill: any → stop and fix
stage: 8 to 10

edit_distance — guardrail

Median edit distance

unit: draft
formula: median normalized edit distance between draft and what was posted
window: until decided · lower is better
events: drafts.edit_distance
kill: rises 50%+ over control
stage: 10

illustrative example

From the waitlist goal to the metrics a test can use.

Joins are the north star, but at that volume no single test can decide on them. So the goal breaks into drivers a test can move, each with its own metric, and guardrails that keep the drivers honest.

Level	Metric	Role in tests	Why
North star	waitlist joins per week, toward the goal	reported, rarely decisive	too few per week for any one test to move visibly
Driver	waitlist offers made per week	diagnostic on the funnel	the constraint: offers weren't going out, so joins can't happen
Driver	signed_up_14d per lead offered	primary for offer tests, when volume allows	the conversion step a campaign can change
Driver (proxy)	replied_7d per lead	primary for first-move and opener tests	ten times more frequent than joins; validate it against signups
Driver	waitlist_join_rate per visitor	primary for page tests	posts and replies send visitors; the page converts them
Driver	cleared_150_24h and resonance per 100 impressions	primary for post tests	reach and attention feed visits
Guardrails	stop requests, negative feedback, founder minutes, tool spend, policy violations	veto on every test	a lift bought with spam or burnout is not a win

failure modes

How metrics mislead, and how to catch it.

Redefined at readout — "Replies" quietly becomes "positive replies" after the data comes in. · tests store the metric version; the readout refuses a definition other than the registered one.

Calendar windows — Units exposed late in the test get less time, and whichever arm sent later loses. · windows are measured per unit from exposure; open windows are pending.

Wrong unit in the ratio — Replies per touch with lead-level randomization gives intervals that are too narrow. · per-unit binary metrics by default; ratio metrics require the delta method, checked at registration.

Vanity primary — Impressions go up while replies and joins don't, and the test is called a win. · impressions-only primaries need a secondary on replies or clicks, and a guardrail on negative feedback.

Proxy drift — Replies rise but signups don't follow, and decisions keep being made on replies. · every lead test records both; the learnings base tracks reply lift vs signup lift across tests.

Novelty read as a win — A new format shines in week one and fades by week three. · first-half vs second-half comparison at every readout; long enough tests for post formats.

Gaming the metric — Openers drift toward bait because bait clears 150 impressions. · negative feedback and edit-distance guardrails, plus the content rules that ban bait outright.

data to capture

What the metric layer records, and what that trains.

Record — every metric definition and version, append-only; per-unit values frozen at window end; per-arm aggregates and intervals at readout; every guardrail check and kill

Label — per-unit outcomes are the labels models learn from; guardrail hits are labels for what not to do

Trains — the proxy map (how replies predict signups); early-read models (does a post's 30-minute snapshot predict its 24-hour threshold?); risk models for stop requests

Test — the metric layer itself: share of tests that finish with no definition change; share of readouts blocked by a guardrail or diagnostic