phase 4 · learn

Experimentation engine

Go deeper

Every stage of the pipeline has a knob: a prompt, a rule, an offer, an opener, a send time. The experimentation engine decides which knob to test, assigns units at random and logs the odds, reads the result against a rule written before launch, and records what it learned. This page defines the experiment as an object, catalogs the knobs, and lists ready-to-run tests.

knob → experiment · the hypothesis and its rule → assignment · arm + odds → exposure · the unit saw it → readout → learning

rule 1 — One live test per knob. · A unit is never in two tests on the same knob, or neither result means anything.

rule 2 — Size it before launch. · A small account can only detect big effects. Test big changes, and write the sample size down first.

rule 3 — Read out on schedule, win or lose. · Every readout goes into learnings, including the losses and the inconclusive ones.

what it is, in industry terms

An experimentation platform, sized for one founder.

Product teams call this an experimentation platform or A/B testing framework. The vocabulary is the same at any size; what changes is how big an effect you can see.

Hypothesis — One sentence: changing this knob moves this metric, because of this reason.

Unit — unit of randomization · The thing that gets an arm: a post, a lead, a visitor, an account, a day.

Arm — variant · control · One version of the knob.

Holdout — A share of units that gets nothing at all, so the effect of acting is measured, not assumed.

Propensity — assignment probability · The odds a unit had of landing in its arm. Logged so old decisions can judge new policies.

Exposure — The moment a unit actually saw its arm. Assigned but never shown doesn't count.

Primary metric — The one number the win rule is about. One per test.

Guardrail — guardrail metric · A number that must not get worse: stop requests, spam reports, founder minutes, tool spend.

MDE — minimum detectable effect · The smallest change the test can see at its sample size. Below it, the result is noise.

Win rule — pre-registration · The decision rule written before launch, so the readout can't be argued into a win.

Shadow mode — The new version runs beside the live one on the same inputs; only the live one acts.

Off-policy evaluation — Estimating how a new policy would have done, from old decisions and their logged odds.

the seven parts

Hypothesis registry — One row per test: the constraint it targets, the knob, the hypothesis, the primary metric, guardrails, sample size, win rule and judge-by date. All set before launch.
Assignment — The unit, the arms and their shares, a holdout, the same arm every time for a unit, and the probability logged on every assignment.
Exposure log — One row per post, message, lead or visit in a test, with its arm and the ids that join it to outcomes, such as the touch_id.
Metrics — Each metric defined once, in code (replies per touch within 7 days, say), plus guardrails: stop requests, spam reports, founder minutes, tool spend.
Readout — Runs on the judge-by date, checks that the sample size was reached, and applies the rule set before launch. No verdicts from peeking.
Decision and learnings — Ship, revert or extend, written to the learnings log with the effect size and how sure it is. Losses count too.
Next test — The learnings set the priors; the current constraint, the weakest step of the funnel, picks the next knob.

the experiment object

Every test is one record, written before launch.

The experiment row is the pre-registration: everything that decides the verdict is filled in before the first unit is assigned. The readout fields are the only ones written later.

Field	Type	What it holds
`experiment_id`	string	Stable and readable: `exp.<stage>.<knob>.<n>`. Every unit and readout carries it.
`hypothesis`	text	One sentence: changing the knob moves the metric, and why.
`constraint`	funnel step	The weakest step this test is meant to move. Tests that target no constraint wait.
`stage`	1 to 12	Which pipeline stage the knob belongs to.
`knob`	string	The one thing that differs between arms: `opener`, `judge_prompt`, `offer`.
`unit`	post \| lead \| visitor \| account \| day \| stored item	What gets an arm. Decides what the sample size counts.
`arms`	list	Each arm's name, what it changes, and its share, e.g. control 45%, variant 45%, holdout 10%.
`holdout`	share	Units that get nothing. Required for campaign tests; zero is allowed elsewhere.
`assignment`	method + logging	Random, sticky per unit, block-randomized or replayed; the probability is written on every unit row.
`primary_metric`	metric id	One metric, defined once, with its window: `replied_7d`, `impressions_24h`.
`guardrails`	metric ids + limits	Must not get worse: stop requests, spam reports, founder minutes, tool spend.
`min_per_arm`	integer	Sample size per arm, from the sizing rule, fixed before launch.
`win_rule`	text	The verdict rule, written in advance: "variant wins if it clears the threshold in 2 of 3 blocks and no guardrail breaks".
`judge_by`	date	When the readout runs, whether or not the result looks interesting.
`status`	draft \| live \| stopped \| read	Where the test is. Only `draft` rows can be edited.
`readout`	snapshot	Counts per arm, the metric per arm, sample sizes and guardrails, frozen at judge-by time.
`decision`	ship \| revert \| extend	What was done, with the new version string if something shipped.
`learning_id`	learning id	The learnings-log row this readout produced.

one experiment, as data

{
 "experiment_id": "exp.sequence.opener.01",
 "hypothesis": "A proof opener (a number from your own work) earns more replies than a plain claim, because it reads as evidence, not opinion.",
 "constraint": "reached → engaged",
 "stage": 10,
 "knob": "opener",
 "unit": "post",
 "arms": [
 {
 "name": "control",
 "change": "plain claim opener",
 "share": 0.5
 },
 {
 "name": "proof",
 "change": "proof opener",
 "share": 0.5
 }
 ],
 "holdout": 0,
 "assignment": {
 "method": "block-randomized",
 "blocks": "3 weeks",
 "p_assign_logged": true
 },
 "primary_metric": "impressions_24h >= 150",
 "guardrails": [
 "negative_feedback = 0",
 "founder_minutes <= 20/day"
 ],
 "min_per_arm": 6,
 "win_rule": "proof wins if both its posts in each block clear 150 impressions at 24h and the control's don't; otherwise inconclusive",
 "judge_by": "YYYY-MM-DD",
 "status": "draft",
 "readout": null,
 "decision": null,
 "learning_id": null
}

the experiment unit

One row per post, lead, visitor or message in a test.

This is the exposure log. It joins the arm to outcomes through the same ids the rest of the pipeline uses, so a readout is a query, not a spreadsheet.

Field	Type	What it holds
`unit_id`	string	The post id, lead id, visitor id or stored item id.
`experiment_id`	string	Which test it belongs to.
`arm`	string	The arm it was assigned: `control`, `variant`, `holdout`.
`p_assign`	0 to 1	The probability it had of getting this arm. Required for off-policy estimates later.
`assigned_at`	timestamp	When the arm was drawn.
`exposed_at`	timestamp	When the unit actually saw its arm. Null if it never did.
`ids`	object	Join keys into the pipeline: `touch_id`, `lead_id`, `post_id`, `signal_id`.
`outcome`	object	Filled at readout from events: the primary metric and each guardrail for this unit.
`block`	string	For block-randomized tests: which block of days the unit ran in.

one unit, as data

{
 "unit_id": "post_9f1a",
 "experiment_id": "exp.sequence.opener.01",
 "arm": "proof",
 "p_assign": 0.5,
 "assigned_at": "2026-10-19T12:04:00Z",
 "exposed_at": "2026-10-19T12:04:31Z",
 "ids": {
 "post_id": "post_9f1a",
 "touch_id": null
 },
 "outcome": {
 "impressions_24h": null,
 "replies_24h": null
 },
 "block": "week 1"
}

five ways to run a test

Pick the mode by stage.

Changes on the listen side (brief, ICP, detectors, the judge, the scorer) can be replayed over stored data with no live risk. Changes on the act side touch real people, so they need real randomization.

Offline replay

Re-run the new version over stored raw items and their labels. No live risk.

brief prompt, ICP lines, judge, scorer

Shadow

The new version runs beside the live one on the same items; compare verdicts, then switch.

detectors and lanes

Randomized A/B

Random assignment with logged odds, a sample size fixed in advance, one knob at a time.

drafts, openers, send times, offers, pages

Holdout lift

A share of eligible units gets no touch, so the effect of acting is measured instead of assumed.

campaigns

Off-policy estimate

Judge a new policy on old decisions using their logged odds, before it touches a real lead.

the playbook recommender

knob catalog

Every stage has something to test.

One row per stage: the knob, how it's tested, the unit, and the metric. The same list appears as the "test" line on each stage on the pipeline overview.

Stage	Knob	Mode	Unit	Metric
1 · Link	hero copy, intake order (login before or after the brief)	A/B by visitor	visitor	finished intake; signup
2 · Brief	extractor prompt and model	offline replay on stored sites, scored against founder corrections	stored site	fields kept unedited
3 · ICP	which fit and not-if lines are in	offline: rescore past leads against accept and paid labels	past lead	agreement with accept and paid
4 · Trigger	which triggers are on; each freshness window	offline from the decay curve, then by lead cohort	signal / lead	reply rate by signal age
5 · Detector	rule, vocabulary, source	shadow: the new version runs beside live on the same items	raw item	matches labeled real per 100
6 · Signal	judge prompt and model	offline replay against founder labels	stored match	precision at the top of the daily list
7 · Lead	scorer weights, tier cutoffs	offline against reply labels, then a randomized slice of the daily top 10	lead	replies among the daily top 10
8 · Playbook	which play to run	off-policy estimate from logged odds, then a live exploration slice	move	reply or signup per move
9 · Campaign	offer, stop rule, holdout share	randomized arms with a holdout	lead	signup lift against the holdout
10 · Sequence	opener, hook, length, proof, send time, number of steps	randomized per draft (prompt version and slot)	draft / post	replies, impressions at 24h
11 · Funnel	landing and waitlist copy, the call to action	A/B by visitor	visitor	waitlist join rate
12 · Loop	review cadence, which constraint goes first	not randomized; tracked as learnings	week	constraint moved by the pre-set rule

the library

Ready-to-run tests.

Each card is a complete experiment record minus the dates. Copy one, fill in the judge-by date, and it's ready to launch. Sample notes assume a small account; see sizing below.

exp.sequence.opener-proof — Randomized A/B

Proof opener vs plain headline

stage 10 · act

if: A number from your own work in the first line earns more reach and replies than a plain claim.
unit: post
arms: control: plain claim · variant: proof opener, same body
metric: impressions at 24h; replies
guard: negative feedback
size: Pairs with a threshold rule.

exp.playbook.reply-first — Randomized A/B

Reply first vs DM first

stage 8 · act

if: Answering in public before a DM earns more replies than a cold DM on a fresh signal.
unit: lead
arms: control: DM first · variant: public reply, DM only after they answer
metric: replied within 7 days
guard: stop requests, blocks
size: About 144 leads per arm to see 5% → 15%.

exp.sequence.send-time — Randomized A/B

Best hour vs off hour

stage 10 · act

if: Posts in the account's best measured hour beat the same kind of post three hours later.
unit: post
arms: control: off hour · variant: best hour from the engagement matrix
metric: engagement score at 24h
guard: none
size: Randomize the slot per draft; keep topics balanced across arms.

exp.signal.judge-v2 — Offline replay

Judge prompt v2

stage 6 · listen

if: A judge prompt with three worked examples agrees with founder labels more often.
unit: stored match
arms: v1 verdicts vs v2 verdicts on the same labeled matches
metric: precision on the top 20 a day
guard: recall must not drop
size: Needs about 200 founder-labeled matches. Free to run; no live risk.

exp.detector.vocab-v2 — Shadow

Detector vocabulary v2

stage 5 · listen

if: Adding buyer phrases from replies raises real matches without raising noise.
unit: raw item
arms: live v1 · shadow v2 on the same feed
metric: matches labeled real per 100
guard: run cost per real match
size: Two weeks of shadow, then a hand check of 50 disagreements.

exp.trigger.fresh-window — Offline replay

Freshness window 24h vs 72h

stage 4 · listen

if: Signals older than 24 hours reply far less, so the window should be shorter.
unit: signal
arms: bucketed by age at first touch: 0–24h, 24–72h
metric: reply rate by age bucket
guard: none
size: Read the decay curve offline first; then randomize which cohort gets touched late.

exp.campaign.offer-brief — Randomized A/B

Offer: free brief vs waitlist

stage 9 · act

if: Offering a free product brief first gets more signups than asking for the waitlist directly.
unit: lead
arms: control: waitlist link · variant: free brief, waitlist after
metric: signed up within 14 days
guard: stop requests; founder minutes
size: Signups are rare: test only with a big expected gap, or use replies as a proxy.

exp.campaign.holdout-lead — Holdout lift

Holdout lift on Lead to waitlist

stage 9 · act

if: The Lead-to-waitlist campaign causes signups that wouldn't happen anyway.
unit: lead
arms: 90% treatment · 10% holdout gets no touch
metric: signup rate, treatment minus holdout
guard: none
size: Needs the most units of any test. Run it continuously and read it monthly.

exp.playbook.comment-kw — Randomized A/B

Comment a keyword vs link in reply

stage 8 · act

if: "Comment X and I'll send it" earns more replies and joins than a link under the post.
unit: post
arms: control: link in reply · variant: comment-a-keyword
metric: replies per post; joins per post
guard: replies you can't answer within a day
size: Alternate in randomized order; 6 to 10 posts per arm finds only a large gap.

exp.sequence.article-long — Randomized A/B

X Article vs long post

stage 10 · act

if: The Article format reaches further than the same text as a long post.
unit: piece of content
arms: control: long post · variant: X Article, same text
metric: impressions at 24h; profile visits
guard: none
size: Same text in both arms isolates the format, not the topic.

exp.sequence.amplify — Randomized A/B

Plain repost vs quote

stage 10 · act

if: A quote from the founder account lifts the product account's post more than a plain repost.
unit: product-account post
arms: control: nothing · repost · quote
metric: engagements at 24h on the product post
guard: none
size: Block-randomized: each arm once per 3-day block. A first run can use as few as 3 blocks.

exp.lead.scorer-v2 — Offline replay

Lead scorer v2

stage 7 · listen

if: Weighting signal freshness higher puts more repliers in the daily top 10.
unit: lead
arms: v1 rank vs v2 rank over past leads with known replies
metric: replies among the top 10 a day
guard: no tier inflation
size: Offline first on frozen features; then a randomized half of the daily list.

exp.playbook.recommender — Off-policy estimate

Playbook recommender v1

stage 8 · act

if: A learned play choice beats the hand-written default on reply rate.
unit: move
arms: current policy vs new policy, estimated on logged moves
metric: estimated reply rate
guard: contact-policy violations = 0
size: Impossible until move odds are logged. Then estimate before any live slice.

exp.icp.icp-cut — Offline replay

Cut a weak ICP line

stage 3 · understand

if: Dropping a fit line that never predicted payment keeps accuracy and grows the list.
unit: past lead
arms: with the line vs without it
metric: agreement with founder accept and with paid
guard: precision must not drop
size: Runs on the reviewed leads you already have.

exp.brief.brief-v2 — Offline replay

Brief extractor v2

stage 2 · understand

if: A second pass that quotes the source for every field reduces founder corrections.
unit: stored site
arms: v1 brief vs v2 brief on the same sites
metric: fields kept unedited
guard: cost per brief
size: Every corrected field is a test case; about 50 is enough to start.

exp.funnel.hero-copy — Randomized A/B

Waitlist page hero copy

stage 11 · learn

if: Naming the pain in the headline beats naming the product.
unit: visitor
arms: control: product headline · variant: pain headline
metric: waitlist join rate
guard: bounce rate
size: Visitor tests need traffic: about 1,164 visitors per arm to see 2% → 4%.

exp.sequence.steps-2v4 — Randomized A/B

Two steps vs four steps

stage 10 · act

if: A shorter sequence loses few replies and cuts stop requests.
unit: lead
arms: control: 4 steps · variant: 2 steps
metric: replied within 14 days
guard: stop requests
size: Read the guardrail as closely as the metric.

sizing for small accounts

How many units each arm needs.

For a yes or no metric (replied, joined, cleared a threshold), a common rule of thumb for 80% power at 5% significance is n per arm ≈ 16 · p(1 − p) / d², where p is the average rate across the two arms and d is the difference you want to see. Small accounts can only see big differences, so test big changes.

Metric	Control	Arm	Units per arm	What that means
Replied within 7 days	5%	10%	444 leads	A doubled reply rate still needs hundreds of leads per arm.
Replied within 7 days	5%	15%	144 leads	Tripling it is visible in weeks of daily outreach.
Cleared a threshold at 24h	30%	50%	96 posts
Replied (any)	10%	20%	204 units	A common size for post-level reply tests.
Waitlist join per visitor	2%	4%	1,164 visitors	Page tests need real traffic; replay or proxies first.

When the numbers are too small

Pick a metric with a higher base rate. Profile clicks or replies per post move more often than signups.
Use a threshold rule. "Clears 150 impressions at 24 hours" turns a noisy count into a yes or no per post.
Replay instead of waiting. Anything on the listen side can be tested offline against stored labels today.
Block by time. Run each arm once per block of days, so a slow week hits every arm equally.
Pool across accounts later. One founder's data is small; the same test across many accounts is not.

The unit is the post, not the impression

Impressions on one post rise and fall together, so a thousand impressions on one post is one unit, not a thousand. Count posts, leads or visitors: whatever was assigned the arm.

The same goes for leads in a campaign: one lead is one unit, however many messages it gets.

the learnings log

What the engine remembers.

A readout is a fact about one test. A learning is what that fact means for the next decision. One learning per readout, linked to it, and rechecked on a date so old learnings don't harden into folklore.

Field	Type	What it holds
`learning_id`	string	Stable id, linked from the experiment row.
`experiment_id`	string	The test that produced it.
`claim`	text	What was learned, in one sentence a founder could act on.
`effect`	number + interval	The size of the difference per arm, with its range.
`confidence`	strong \| suggestive \| inconclusive	Set by the win rule, not by how the chart looks.
`scope`	channel, account size, stage	Where it was learned, so it isn't applied where it wasn't.
`shipped`	version string	The change that went out because of it, if any.
`recheck_on`	date	When to run it again before trusting it further.
`evidence`	links	The readout snapshot and the units behind it.

rules for the log

Losses count. "The quote didn't beat the repost" is as useful as a win.
Inconclusive stays inconclusive. Don't promote a near-miss to a finding.
Scope it. A learning from one channel and account size is labeled with both.
Recheck it. Platforms change. Every learning carries a recheck date.
It sets the prior. The next test on the same knob starts from it.

how it connects

Every stage feeds it; stage 12 owns it.

The engine reads the events every other page defines and writes back into the knobs those pages describe.

GTM tracking plan — The event catalog the engine reads. Its assignment and exposure events (campaign arm, assignment probability) are this page's unit rows.

Playbook Library — Where the act-side knobs live: openers, offers, steps and stop rules.

Detector catalog — Detector versions are what shadow tests compare.

Buying Moments Map — Triggers and their freshness windows, the stage 4 knobs.

Pipeline — Stage 12 on the canvas: the loop that sends learnings back into triggers and detectors.

data to capture

What the engine records, and what that trains.

Record — every assignment with its arm and probability; every exposure with a timestamp; the readout frozen at the judge-by date; each change with what, where, the new version and the win rule set before; a daily snapshot of counts per rung, trigger and detector

Label — did the change move the constraint, by the rule set beforehand; strong, suggestive or inconclusive

Trains — a learnings base that sets priors for the next test; off-policy estimates for the recommender; later, an agent that proposes the next change

Test — review cadence and which constraint goes first: not randomized, tracked as learnings