learn · experimentation engine

Learnings base

Every readout, win or lose, and the priors it sets.

In Experimentation engine

A readout is a fact about one test. A learning is what that fact means for the next decision. The learnings base keeps one row per readout, wins and losses and inconclusives alike. Each row records how sure it is and where it applies. It's what turns the next test's prior from a guess into evidence, and it's what an agent reads before proposing anything new.

readout → learning · claim, effect, confidence, scope → prior → next test proposed → registry

what it is, in industry terms

An experiment repository with meta-analysis on top.

Large experimentation programs keep a searchable archive of every test and its result, so teams stop re-running old tests and can pool related ones (a meta-analysis). Without it, results live in someone's memory, then harden into folklore ("quotes always beat reposts") long after the platform changed. For one founder the archive is small, so each row has to be honest about how much it can carry.

Learning — One sentence a founder could act on, tied to the readout that supports it.

Effect size — How big the difference was, with its interval. Not just "won".

Confidence — strong · suggestive · inconclusive · void · Set by rule from the readout, not by how exciting the result feels.

Scope — Where it was learned: account, channel, audience size, stage, date range.

Prior — What the next test on the same knob starts from, discounted by scope.

Meta-analysis — Pooling several tests of the same knob into one estimate.

Winner's curse — Winning tests overstate their effect; replications come in smaller.

Recheck date — When a learning must be retested before anyone leans on it harder.

how it works

From readout to the next test.

Every readout writes exactly one learning row, even when the answer is "we can't tell". Rows are tagged by stage, knob, channel and account. When the constraint picks a knob for the next test, the matching rows set its prior and its sizing, and an agent drafts the experiment record for the founder to approve.

Readout — frozen at the judge-by date
Learning — row claim, effect + interval, confidence
Tags — stage, knob, channel, account
Prior — discounted by scope and confidence
Constraint — picks a knob the weakest funnel rung
Agent drafts — a test sized from the prior
Founder — approves then the registry takes over

confidence, by rule

Four labels, each with a test it must pass.

Label	Requires	Can be used for	Prior strength it lends
strong	sample reached; pre-set rule passed; the rule's false-win rate is 5% or less; no unresolved confound; or a suggestive result replicated by its own pre-set rule	shipping a default and building on it	full: confidence weight 1.0
suggestive	the rule passed, but its false-win rate is above 5%, or there's a known confound or timing problem, or the units were few	shipping a cheap default and scheduling a replication	half: 0.5
inconclusive	sample not reached, or the interval spans zero, or one unit per arm	noting a direction to test properly	a fifth: 0.2
void	sample ratio mismatch, broken exposure, an arm never ran	nothing: fix and rerun	none: 0

the cap rule

A learning's confidence can never be higher than its win rule allows. If the rule declares a winner by luck more than 5% of the time, the result is at most suggestive, however lopsided it looks (see the false-win rates on the readout page). A replication that passes its own pre-set rule can lift a suggestive learning to strong.

priors for the next test

Discount past results by how far away they were learned.

A past learning becomes a Beta prior for the next test on the same knob, the same way the readout page builds one. The prior's strength is the past sample size, shrunk by a scope weight and capped, so one old test can't overpower new data.

prior mean m = the past arm's observed rate (or the pooled rate across matching learnings) prior strength s = min( cap , scope_weight × confidence_weight × past units ) prior Beta( m · s , (1 − m) · s ) scope_weight: same account and channel 0.5 · same account, other channel 0.2 · other account 0.1 confidence_weight: strong 1.0 · suggestive 0.5 · inconclusive 0.2 · void 0 cap: 50 pseudo-units

worked example

A past arm observed a 30% rate over 40 units, in the same account and channel, with suggestive confidence. Strength = min(50, 0.5 × 0.5 × 40) = 10 pseudo-units.

the prior the next test starts from

Beta(3.0, 7.0) — the prior

10.0 — pseudo-units of weight (cap 50)

10% to 55% — 90% range before new data

The 90% range shows how much the prior constrains the next test before any new data. A wide range means the new test decides; a narrow one means the past still weighs in.

the learning object

One row per readout, never deleted.

Learnings are append-only. When a recheck overturns one, a new row supersedes it and the old row stays, so the history of what was believed, and when, is kept.

Field	Type	What it holds
`learning_id`	string	`ln.<stage>.<knob>.<n>`
`experiment_id`	string	The test that produced it.
`readout_id`	string	The frozen readout it rests on.
`claim`	text	One sentence a founder could act on.
`effect`	object	Per-arm values, the difference and its interval, in the metric's own units.
`n`	per arm	Units behind the effect.
`rule`	object	The pre-set rule, its result, and its false-win rate under no difference.
`confidence`	strong \| suggestive \| inconclusive \| void	From the confidence table, by rule.
`caveats`	list	Confounds, timing problems, isolation problems.
`scope`	object	Account, channel, audience size at the time, stage, knob, dates.
`tags`	list	For retrieval: stage, knob, metric, play, format.
`shipped`	version string, optional	What changed because of it.
`prior_out`	object	The Beta prior this learning lends to the next test on the same knob.
`recheck_on`	date	When to retest before relying on it further.
`supersedes`	learning id, optional	The older learning this one replaces. Old rows are kept.

one learning, as data (illustrative)

{
  "learning_id": "ln.post.opener.01",
  "experiment_id": "exp.post.opener.01",
  "readout_id": "ro.exp.post.opener.01.1",
  "effect": {
    "metric": "impressions_24h >= 150",
    "proof": "2 of 2 (310, 188)",
    "plain": "0 of 2 (74, 51)",
    "secondary": {
      "replies": {
        "proof": 14,
        "plain": 3
      }
    }
  },
  "n": {
    "proof": 2,
    "plain": 2
  },
  "rule": {
    "text": "both posts in a version clear 150 impressions at 24h and the other version's don't",
    "result": "pass: proof",
    "false_win_rate_max": 0.125
  },
  "confidence": "suggestive",
  "caveats": [
    "metrics not pulled at exactly 24h",
    "other posts went out the same week",
    "the post bodies differed, so the opener wasn't perfectly isolated"
  ],
  "scope": {
    "channel": "x",
    "stage": 10,
    "knob": "opener",
    "dates": "2026-05-05/2026-05-12"
  },
  "tags": [
    "opener",
    "proof",
    "list post"
  ],
  "shipped": "proof opener as default (proposed)",
  "prior_out": "Beta built from the proof arm, weight 0.5 × 0.5 × units",
  "recheck_on": "after a replication on another channel",
  "supersedes": null
}

Illustrative values, invented to show the shape.

illustrative example

Constraint first, then the knob with the most to learn per unit.

Read the constraint — Find the weakest funnel rung this week from the daily snapshot.
List the knobs — Every knob at the stages that feed that rung, from the knob catalog.
Pull the learnings — For each knob: matching rows by tags, their effects, confidence and scope.
Score each knob — Priority = uncertainty × plausible lift × reach ÷ (units needed × risk).
Prefer cheap modes — Replay or shadow if the knob allows it; live only when it can't be tested offline.
Draft the record — Hypothesis, unit, arms, prior, sample size, win rule and judge-by date, all filled in.
Ask the founder — The draft goes to the founder; nothing launches without approval.

priority(knob) = uncertainty × plausible lift × reach ÷ (units needed × risk) uncertainty = width of the current prior's interval for that knob (wide = more to learn) plausible lift = the effect size the learnings suggest is possible (shrunk for winner's curse) reach = share of daily units the knob touches units needed = from the sizing rule at that lift; replay-able knobs count as nearly free risk = 1 for replay, 2 for shadow, 3 for live with guardrails, 5 if it touches strangers

Reach exists (posts get impressions); conversion to sign-up is the weak rung.
A past test suggests the post format matters more than the length of a list. A second test on a related knob is running, and its win rule is loose enough to pick a winner by luck.
Proposal. Replicate the proof opener on posts that carry the sign-up link, so the test moves the weak rung directly. A replication can lift the learning to strong.
Also proposed. Before that test's readout, add a stricter secondary rule (3 of 3 blocks) so its result can't be over-read.

failure modes

How a learnings base rots, and how to catch it.

Folklore — An old suggestive result gets repeated until it's treated as law. · Every row carries a confidence and a recheck date; past-due rows show as stale.

Only wins get written — Losses and inconclusives are skipped, so priors lean optimistic. · A readout can't close without its learning row, whatever the verdict.

Winner's curse — The shipped effect is smaller than the test's. · Shrink effects toward the prior; schedule a replication for anything shipped.

Confounded learning — A result driven by a viral week is filed as a headline effect. · Caveats are required, and a known confound caps confidence at suggestive.

Duplicate tests — The same knob is tested again without reading the last result. · The agent's proposal step pulls matching learnings by tag first.

Rewriting history — An old learning is edited when a recheck disagrees. · Append-only: new rows supersede old ones; old rows stay.

Over-reading a ranking — A topic ranking from two probes per lane is filed as a winner. · Rankings are logged as directions; a 2-arm test confirms the top pick.

data to capture

What the learnings base records, and what that trains.

Record — one learning per readout with claim, effect and interval, n per arm, the rule and its false-win rate, confidence, caveats, scope, tags, what shipped, the prior it lends, a recheck date and what it supersedes

Label — the confidence label by rule; later, whether a replication confirmed or overturned it

Trains — priors for every next test; a calibration of how often suggestive learnings replicate; an agent that proposes and sizes the next test

Test — the base itself: replication rate by confidence label. If suggestive results rarely replicate, the rules are too loose.