learn · experimentation engine

Hypothesis registry

One row per test, everything set before launch.

In Experimentation engine

The registry is the one place every test is written down before it starts: what changes, who gets it, what number decides it, what must not get worse, how many units it needs and the date it will be read. Once the first unit is assigned, the parts that decide the verdict are frozen. That is what turns a test from an opinion with a chart into a decision you can trust.

registry · write the rule first → assignment → exposure log → metrics → readout → learning

rule 1 — Nothing runs unregistered. · If a knob changes for some units and not others, it is a test, and it gets a row before the first unit sees it.

rule 2 — The verdict rule is frozen at launch. · The metric, its window, the win rule and the sample size can't change after the first exposure. Amendments are logged, never silent.

rule 3 — One live test per knob, per population. · Two tests that change the same thing for the same units are blocked at registration, not discovered at readout.

what it is, in industry terms

Pre-registration, kept as a table.

Clinical trials call it pre-registration. Product teams call it an experiment registry or experiment review. The job is the same: commit to the question, the measurement and the decision rule before the data can tempt you. On a small account this matters more, not less, because every result is noisy and it is easy to find a story in noise.

Pre-registration — Writing the hypothesis, metric, sample size and decision rule before any unit is exposed.

HARKing — hypothesizing after results are known · Picking the hypothesis that fits the data you already saw. The registry makes it visible.

Forking paths — Many small analysis choices (which metric, which window, which segment) that together can turn noise into a "win".

Layer — mutually exclusive layer · A group of tests that can't share units, usually one per knob. Tests in different layers may overlap.

MDE — minimum detectable effect · The smallest difference the sample can see. Set before launch from the base rate.

Judge-by date — When the readout runs, whatever the numbers look like that morning.

Kill threshold — A guardrail limit that stops the test early, written down in advance.

Amendment — A logged change to a registered test, with a reason. Allowed only for fields that don't decide the verdict.

how it works

A test moves through six states, and each move has a gate.

A test can only move forward when the gate's checks pass. The checks are mechanical: a field is filled or it isn't, a conflict exists or it doesn't. That's what lets an agent propose tests later without being able to quietly move the goalposts.

draft — editable
registered — verdict fields frozen
running — units assigned
readout — snapshot frozen
decided — ship · revert · extend
learning — sets the next prior gate A gate B gate C gate D gate E
stopped — guardrail or bug still read out, labeled stopped
abandoned

Gate	Move	What must be true	What gets frozen
gate A	draft → registered	Every required field is filled; the hypothesis fills all seven brackets; baseline has a source window; units per arm is computed from baseline and MDE; at least one guardrail with a kill threshold; the win rule names the metric, the threshold and the judge-by date; no running test in the same layer with an overlapping population; the founder signs off.	hypothesis, unit, population, arms and shares, primary metric and window, guardrails and kill thresholds, units per arm, win rule, analysis plan. A hash of these fields is stored as `registered_hash`.
gate B	registered → running	The assignment salt is set; one smoke unit was assigned and its exposure event arrived; assignment probability is being logged on unit rows.
gate C	running → readout	The judge-by date arrived, or every arm reached its units per arm, whichever the win rule names. Outcome windows for the counted units have closed.	the readout snapshot: counts per arm, metric per arm with its interval, guardrails, sample ratio check, attrition per arm.
gate C′	running → stopped	A guardrail crossed its kill threshold, or a bug was found in assignment or exposure. The reason is required.	the stop reason and the units so far. A stopped test still gets a readout, labeled stopped.
gate D	readout → decided	The win rule is applied as written. Extend is allowed only if the registration said so, and only once.	the decision, the version string that shipped (if any), and the hash check: the rule read equals `registered_hash`.
gate E	decided → learning	A learning row is written, even for a loss or an inconclusive result.	`learning_id` on the experiment row.

the object

The registered experiment, field by field.

This extends the experiment object on the parent page with what pre-registration needs: a baseline with its source, the population, the analysis plan, kill thresholds, a layer for the conflict check, and a hash that proves the registered rule is the one that was read.

Field	Type	What it holds
`experiment_id`	string	`exp.<stage>.<knob>.<n>`, never reused.
`title`	text	A short human name: "Free brief vs waitlist link".
`owner`	person or agent	Who registered it. An agent may draft; a founder signs off at gate A.
`hypothesis`	text	The seven-bracket sentence: knob, control, variant, population, metric, MDE and window, mechanism.
`constraint`	funnel step	The weakest step it targets, e.g. `waitlist offered → signed up`.
`stage · knob`	1 to 12 · string	Where the knob lives. Together with the population, this makes the layer.
`layer`	string	`<stage>.<knob>`, or a shared layer such as `account.posting_calendar` for things that share an audience.
`baseline`	rate + source	The control's current rate and where it was measured: `replied_7d = 0.05, leads touched Sep 20 to Oct 16`.
`unit · population`	enum · rule	What gets an arm, and which units are eligible: "Hot and Warm leads with a pain signal under 72 hours old".
`arms · holdout`	list · share	Each arm's change and share; the holdout share (0 allowed except for campaign tests).
`assignment`	method	Sticky hash, block-randomized, switchback or replay. Details live on the Assignment and randomization page.
`primary_metric`	metric id + window	Exactly one, defined in the metric registry: `signed_up_14d`.
`secondary_metrics`	metric ids	Reported, never decisive.
`guardrails`	metric ids + kill thresholds	Must not get worse, and the line that stops the test: `stop_requests_per_100 > 2 → stop`.
`mde · alpha · power`	numbers	The smallest effect worth seeing, and the error rates. Defaults: 5% and 80%.
`units_per_arm`	integer	From the sizing rule. The readout refuses a verdict below it unless the test is stopped.
`max_duration`	days	When to read out even if the units never arrive. Prevents zombie tests.
`win_rule`	text	The verdict rule, mechanical enough for code to apply.
`analysis_plan`	text	Intention-to-treat by default; exclusions named in advance ("leads that blocked before exposure").
`judge_by`	date	The readout date.
`status`	draft · registered · running · stopped · readout · decided · abandoned	Only draft rows are editable.
`registered_at · registered_hash`	timestamp · hash	When gate A passed, and a hash of the frozen fields.
`amendments`	append-only list	Each change after registration: field, old, new, reason, who, when. Verdict fields can't be amended after the first exposure.
`readout · decision · learning_id`	snapshot · enum · id	Filled by gates C, D and E.

one registered test, as data

{
 "experiment_id": "exp.campaign.offer.01",
 "title": "Free brief vs waitlist link",
 "hypothesis": "If we change the offer from a waitlist link to a free product brief for Hot and Warm leads with a fresh pain signal, then signed_up_14d rises by at least 4 points, because a brief shows the product working on their own site before asking for anything.",
 "constraint": "waitlist offered → signed up",
 "stage": 9,
 "knob": "offer",
 "layer": "9.offer",
 "baseline": {
 "metric": "signed_up_14d",
 "value": 0.02,
 "source": "placeholder: replace with the measured rate before registering"
 },
 "unit": "lead",
 "population": "tier in (Hot, Warm) and pain signal age < 72h and not never_contact",
 "arms": [
 {
 "name": "control",
 "change": "waitlist link",
 "share": 0.45
 },
 {
 "name": "brief",
 "change": "free brief, waitlist after",
 "share": 0.45
 }
 ],
 "holdout": 0.1,
 "assignment": {
 "method": "sticky hash",
 "salt": "exp.campaign.offer.01",
 "at": "eligibility"
 },
 "primary_metric": "signed_up_14d",
 "secondary_metrics": [
 "replied_7d"
 ],
 "guardrails": [
 {
 "metric": "stop_requests_per_100_touches",
 "kill": "> 2 in either arm"
 },
 {
 "metric": "founder_minutes_per_day",
 "kill": "> 30 for 3 days"
 }
 ],
 "mde": 0.04,
 "alpha": 0.05,
 "power": 0.8,
 "units_per_arm": 384,
 "max_duration_days": 60,
 "win_rule": "brief wins if signed_up_14d(brief) - signed_up_14d(control) >= 0.04 with the 95% interval above 0, and no guardrail killed; otherwise keep control",
 "analysis_plan": "intention-to-treat; exclude only leads that became never_contact before assignment",
 "judge_by": "YYYY-MM-DD",
 "status": "draft",
 "registered_at": null,
 "registered_hash": null,
 "amendments": []
}

one amendment, as data

{
 "field": "judge_by",
 "old": "2026-11-20",
 "new": "2026-11-27",
 "reason": "Reddit RSS ends Nov 13; lead volume drops until the replacement source is live",
 "at": "2026-11-10T14:00:00Z",
 "allowed": "yes: judge_by is not a verdict field once max_duration covers it"
}

the hypothesis sentence

If we change [knob] from [control] to [variant] for [population], then [primary metric] moves by at least [MDE] within [window], because [mechanism]. Every row's hypothesis must fill all seven brackets or gate A refuses it.

the conflict check

One live test per knob, per population.

Each test belongs to a layer: the knob it changes, plus the population it changes it for. At gate A, the registry looks for any running test in the same layer whose population overlaps. Different layers can run at the same time if their units don't interfere. On one small account, a few things share an audience no matter what, so they count as one layer.

Test A	Test B	Verdict	Why
		allowed with a condition	Different knobs, same account calendar. Allowed with a factorial or block design; otherwise the second test's posts change the first test's audience.
Free brief vs waitlist (offer, Hot and Warm leads)	Two steps vs four steps (sequence, all touched leads)	allowed	Different layers. A lead can be in both; the readout of each is still clean if assignment is independent (different salts).
Free brief vs waitlist (offer, Hot leads)	Free brief vs call (offer, Hot leads)	blocked	Same knob, same population. Merge into one three-arm test instead.
Judge prompt v2 (offline replay)	Detector vocabulary v2 (shadow)	allowed	Neither acts on people. Replay and shadow never conflict with live tests.
Holdout lift on Lead to waitlist	Any offer test on the same leads	allowed with a condition	Allowed only if the offer test draws from the treatment group, never the holdout. The holdout must stay untouched.
	Amplification quotes (posts, 5 to 8pm)	allowed with a condition	Shared audience; kept clean with reserved time windows. The registry stores those windows as part of the population rule.
Waitlist hero copy (visitors)	Proof opener (posts)	allowed	Different units and layers. Posts drive visitors, so report the hero test by traffic source in case one arm's posts bring a different crowd.
Scorer v2 live slice (daily top 10)	Reply first vs DM first (leads)	allowed with a condition	The scorer decides who is eligible for the reply test. Freeze the scorer version for the reply test's population, or the populations drift mid-test.

a real example of doing this by hand

That is the conflict check, done manually. The registry does the same thing automatically, and it also catches what reserved slots miss: two tests that change the same knob for the same leads.

the library

Twelve registered hypotheses, one per stage.

Each card is a complete registration: the hypothesis sentence, baseline, MDE, units per arm, the win rule and the kill threshold. Sample sizes use the rule of thumb on the parent page (n per arm ≈ 16 · p(1 − p) / d², 80% power, 5% significance). Baselines are placeholders to replace with your own measured rate before registering.

1.intake_order — stage 1

Login after the brief, not before

if: If the Google login moves from before the brief to after it streams, for new visitors, then finished_intake rises by 15 points within the session, because people see value before being asked for anything.
unit: visitor
base: finished_intake ≈ 40% (placeholder)
mde: +15 points
n/arm: 178 visitors
win: variant wins if its rate is 15+ points higher with the interval above 0
kill: bounce rate up 10+ points

2.extractor_prompt — stage 2

Brief extractor quotes every field

if: If the extractor must quote the page sentence behind every field, for stored sites, then fields_kept_unedited rises by 15 points, because unsupported guesses get dropped.
unit: stored site field
base: kept ≈ 60% (placeholder)
mde: +15 points
n/arm: 156 fields
win: v2 wins if kept-unedited is 15+ points higher on the same fields (paired)
kill: cost per brief up 50%+

3.icp_lines — stage 3

Drop the weakest ICP fit line

if: If the fit line with the lowest lift is removed, for reviewed leads, then agreement with founder accept stays within 2 points while the eligible list grows, because the line was filtering noise.
unit: past lead
base: agreement on reviewed leads
mde: non-inferior within 2 points
n/arm: all reviewed leads (about 100)
win: keep the cut if agreement drops by less than 2 points and paid leads are never excluded
kill: any paying customer excluded

4.freshness — stage 4

Freshness window 24h instead of 72h

if: If only signals under 24 hours are touched, for pain signals, then replied_7d rises by 10 points, because the buying window closes fast.
unit: signal
base: replied_7d ≈ 5% (placeholder)
mde: +10 points
n/arm: 144 per age bucket
win: read the decay curve offline first; register the live test only if the offline gap is 10+ points
kill: touched volume drops below 5 a day

5.vocab.reddit_pain — stage 5

Detector vocabulary v2 in shadow

if: If buyer phrases from real replies are added to the vocabulary, for the Reddit pain lane, then real matches per 100 rise by 15 points, because buyers describe the pain in their own words.
unit: raw item
base: real per 100 ≈ 40 (placeholder)
mde: +15 points
n/arm: 178 labeled matches per version
win: v2 wins if precision is 15+ points higher on founder labels and recall doesn't drop
kill: run cost per real match up 50%+

6.judge_prompt — stage 6

Judge with three worked examples

if: If the judge prompt gets three labeled examples, for stored matches, then precision in the daily top 20 rises, because the edge cases are shown, not described.
unit: stored match
base: v1 precision on the same 200 labels
mde: any gain with McNemar p < 0.05
n/arm: 200 labeled matches (paired)
win: v2 wins on the paired disagreements by McNemar's test
kill: recall drops 5+ points

7.scorer_weights — stage 7

Scorer weights freshness higher

if: If signal age gets twice the weight, for scored leads, then replies among the daily top 10 rise, because fresh pain answers.
unit: lead (frozen features)
base: replies in top 10 under v1
mde: +50% relative
n/arm: replay over past scored leads, then 144 leads per arm live
win: replay gain first; live slice wins if replied_7d is 10+ points higher
kill: tier inflation: Hot share up 2x

8.first_move — stage 8

Public reply before any DM

if: If the first move is a public reply instead of a DM, for fresh pain signals, then replied_7d rises by 10 points, because a public answer is helpful before it is a pitch.
unit: lead
base: replied_7d ≈ 5% (placeholder)
mde: +10 points
n/arm: 144 leads
win: variant wins if replied_7d is 10+ points higher with the interval above 0
kill: stop requests > 2 per 100, any block

9.offer — stage 9

Free brief vs waitlist link

if: If the offer is a free brief before the waitlist, for Hot and Warm leads, then signed_up_14d rises by 4 points, because they see it work first.
unit: lead
base: signed_up_14d ≈ 2% (placeholder)
mde: +4 points
n/arm: 384 leads (+10% holdout)
win: brief wins if the gap is 4+ points with the interval above 0
kill: founder minutes > 30 a day for 3 days

account.posting_calendar — stage 10

Proof opener vs plain claim

unit: post
base: clears 150 ≈ 30% (placeholder)
mde: +20 points
win: proof wins if its clear rate is 20+ points higher, blocked by week
kill: negative feedback on any post

11.hero_copy — stage 11

Name the pain in the waitlist headline

if: If the headline names the pain instead of the product, for waitlist visitors, then join rate doubles, because visitors recognize their problem first.
unit: visitor
base: join rate ≈ 2% (placeholder)
mde: +2 points (2% → 4%)
n/arm: 1,164 visitors
win: variant wins if the gap is 2+ points with the interval above 0; report by traffic source
kill: bounce rate up 10+ points

12.review_cadence — stage 12

Twice-weekly review instead of weekly

if: If the loop review runs twice a week, then the constraint moves sooner, because fixes ship faster.
unit: week
base: constraint moved per 4 weeks
mde: not randomized
n/arm: 8 weeks, then compare to the 8 before
win: tracked as a learning; never a causal claim
kill: founder minutes on review > 60 a week

illustrative example

Registering the first offer test for the waitlist goal.

The constraint on the ladder (found → engaged → pain stated → brief offered → waitlist offered → signed up) is late: leads are found and engaged, but offers were not going out. So the first registered act-side test targets the offer itself.

Pick the constraint
The ladder shows leads found and engaged but few offers made, so the knob is the offer at stage 9, layer 9.offer.
Write the seven-bracket hypothesis
Knob: offer. Control: waitlist link. Variant: free brief. Population: Hot and Warm leads with a pain signal under 72 hours. Metric: signed_up_14d. MDE: 4 points. Mechanism: they see it work on their own site first.
Measure the baseline
Replace the placeholder with the measured signup rate of touched leads over the last four weeks. If it can't be measured yet (no touch_id), register replied_7d as the primary and signups as secondary, and say so.
Size it
At a 2% baseline and a 4-point MDE, the rule of thumb gives about 384 leads per arm. That's months at today's volume, so the registration either picks a bigger MDE or the reply proxy (144 per arm at 5% → 15%).
Set guardrails and kill lines
Stop requests over 2 per 100 touches in either arm; founder minutes over 30 a day for three days (a brief takes the founder's time).
Run the conflict check
No other test in 9.offer. The holdout share (10%) is reserved, and the sequence-length test must draw only from treatment leads.
Sign off and freeze
Gate A passes, registered_hash is stored, and the judge-by date goes on the calendar. From here, only the date and notes can be amended.

failure modes

How a registry gets bypassed, and how to catch it.

Moving the goalposts — The metric, window or threshold changes after the data comes in, and the test "wins". · the readout recomputes registered_hash from the rule it applies and refuses a verdict if it differs.

Peeking and stopping early — The test is stopped the day it looks good. Noise early on looks like a big effect. · verdicts only at gate C; an early stop is allowed only through a kill threshold, and is labeled stopped, never "won".

Forking paths — Five metrics and three segments, and one of them is significant. · one primary metric; secondaries and segments are reported, never decisive; the analysis plan names exclusions in advance.

Underpowered "no effect" — Twenty units per arm, no difference, and the idea is declared dead. · units per arm is required at gate A; a readout below it is labeled inconclusive, not negative.

Zombie tests — A test that never reaches its sample and never gets read. · max_duration forces a readout; a weekly job lists running tests past judge-by.

Silent overlap — Two tests change the same knob for the same leads, and neither result means anything. · the layer check at gate A; for account-level tests, a shared layer for the posting calendar.

Unregistered tests — A change goes out to some leads "just to try it". · any unit row whose experiment_id has no registered parent is an error in the daily snapshot.

data to capture

What the registry records, and what that trains.

The registry follows the guide's data rules: log the decision, version whatever decided, freeze the inputs, keep the negatives. Abandoned and stopped tests stay in the table; they are the negatives of the testing process itself.

Record — every registry row and every amendment, append-only, with who and why; registered_hash; gate timestamps; abandoned and stopped tests with their reasons

Label — per test: on-time readout or not; verdict (win, loss, inconclusive, stopped); whether the decision followed the rule

Trains — priors on effect sizes per knob, so the next test picks a realistic MDE; later, an agent that drafts registrations a founder only signs

Test — the registry itself: share of tests read out on time, share stopped by guardrails, share inconclusive. Rising inconclusives mean MDEs are set too small.