learn · experimentation engine

Offline replay and shadow tests

Testing listen-side changes on stored data.

In Experimentation engine

Most changes on the listen side never need to touch a real person to be tested. A new judge prompt, a new scorer, a new detector vocabulary or a new brief prompt can be run over yesterday's stored items and graded against the founder's own labels: that's replay. Or it can run live beside the current version without acting: that's shadow. Both are free of outreach risk, and both only work if the data was recorded the right way from day one.

stored raw items → frozen features → replay · shadow → disagreements labeled → switch or not → learning

what it is, in industry terms

Backtesting and shadow deployment.

Machine-learning teams call replay offline evaluation or backtesting, and shadow mode a shadow deployment or dark launch. Search and recommendation teams use the same pair: score the new ranker offline on logged queries with judged results, then run it in shadow on live traffic before it serves anyone. In VibeDraft, the same idea applies to every component that decides what counts: the brief extractor, the ICP filter, detectors, the match judge, the router and the lead scorer.

Replay — offline evaluation · backtest · Run version B over stored inputs and grade it against stored labels. Nothing live changes.

Shadow — dark launch · Version B runs live beside version A on the same new items. Only A acts; B's verdicts are logged.

Golden set — held-out labeled set · A fixed set of labeled items no prompt was tuned on, kept for the final check.

Disagreement set — Items where A and B decide differently. The cheapest place to spend founder labels.

Precision at k — P@k · Of the top k items a version puts on the daily list, how many are real.

Recall — Of the real items that exist, how many the version catches. Needs kept negatives to estimate.

Label leakage — Grading with information the version couldn't have had at decision time.

Selection bias — Labels only exist for what the old version surfaced, so a new version's new finds are unlabeled.

how it works

Replay first, then shadow, then switch.

Replay answers "would B have done better on what we already saw?" in minutes. Shadow answers "does B hold up on what arrives tomorrow?" over a week or two. Only a version that passes both goes live, and the switch itself is logged as a change with a win rule.

REPLAY · ON STORED DATA

Stored raw items — + frozen features captured, never recomputed
Version A — the live judge
Version B — the candidate
Compare — verdicts agree · disagree
Grade against — founder labels P@20 · recall · κ · cost
Label the — disagreements shrink the unknowns SHADOW · ON NEW ITEMS
New items — from the source
A acts, B logs — same items, same time
One to two weeks — hold the replay result?
Switch — new version stamped
Learning — win or lose

1 · pick the set — Stored items plus labels · A time window of raw items with their frozen features and founder labels. Split off a golden set nobody tunes on.

2 · run both — A and B on identical inputs · Same items, same frozen features, same order. Log every verdict, confidence, cost and latency for both.

3 · label the gaps — Disagreements first · Where A and B agree, labels mostly already exist. Spend founder minutes on the disagreements, plus a small random slice of agreements.

4 · shadow, then switch — One to two weeks live, acting off · B runs beside A on new items. If it holds its replay result and costs fit, switch, and log the change with its version.

why the data rules make this possible

Rule 3 (freeze the inputs) and rule 4 (keep the negatives).

rule 3 · freeze the inputs

A replay is only fair if version B sees exactly what version A saw at decision time. Posts get edited or deleted, author bios change, a lead's later reply leaks into a "fresh" feature. If features are recomputed at replay time, B is graded on information from the future and will look better than it really is.

Store the raw text and author snapshot at capture. The signal page's object keeps both.
Store the scorer's features as JSON at score time, with the scorer version.
Never join outcomes into the input row. Outcomes live in events and are joined only when grading.

rule 4 · keep the negatives

If discarded matches vanish, the only items left are the ones version A already liked. Then you can measure B's precision on A's picks, but not whether B finds real items A threw away. Recall becomes unmeasurable, and every replay is biased toward the current version.

Keep every discard as a row, with its reason: stale, excluded, judge said no, no person.
Label a random slice of discards each week, so recall can be estimated.
Keep items below the list cutoff, not just the daily top 20.

selection bias, and the fix

Founder labels exist almost only for items version A put on the daily list. When B surfaces something A never showed, that item has no label. Treat unlabeled as unknown, not as wrong. Report B's precision as a range: the worst case counts every unlabeled pick as wrong, the best case counts it as right. Then label the disagreements to shrink the range. A switch decision uses the worst case.

grading a replay

Compare A and B on the same labeled items.

From each version's counts on the labeled set, compute precision, recall, F1, agreement with the founder (Cohen's kappa) and cost per real match. The decision usually turns on precision at the top of the daily list, with recall and cost as guardrails.

precision = TP / (TP + FP) recall = TP / (TP + FN) F1 = 2PR / (P + R) kappa κ = (p_o − p_e) / (1 − p_e) p_o = observed agreement with the founder, p_e = agreement expected by chance P@k = real items among the top k a version ranks (k = 20, the daily list)

counts on the labeled set

Count	A	B
True positives	62	71
False positives	38	22
False negatives	18	9
True negatives	82	98
Cost per 100 items	$0.40	$0.65

Example numbers on 200 labeled items. They're illustrative, not real data: swap in a real replay.

A against B

Metric	A	B	Change
Precision	62.0%	76.3%	+14.3 pts
Recall	77.5%	88.8%	+11.2 pts
F1	68.9%	82.1%	+13.2 pts
Agreement with founder (κ)	0.44	0.69	+0.25
Cost per real match	$0.013	$0.018	+$0.005

passes — sample rule: precision +10 points or more, recall not lower

substantial — B's agreement with the founder (κ 0.69)

the objects

A replay run, and a shadow verdict row.

The replay run is the experiment record for offline tests: it names both versions, the exact dataset, the label source and the decision rule, all before running. Shadow verdicts are ordinary rows, one per item per version, so a shadow week can itself be replayed later.

Field	Type	What it holds
`replay_id`	string	`rp.<component>.<n>`. Linked from the experiment record as its unit source.
`component`	brief \| icp \| detector \| judge \| router \| scorer	What is being changed.
`version_a`	version string	The live version, exactly as it ran.
`version_b`	version string	The candidate: prompt version, model, rule set or weights.
`dataset`	object	Source, lanes, time window, item count, and the frozen snapshot id the inputs come from.
`golden_set`	item ids	Held-out labeled items nobody tuned on. Used for the final check only.
`labels`	object	Label source (founder, outcome), count, and when they were made. Labels made after seeing B's output are flagged.
`metrics`	per version	Precision, recall, F1, P@20, κ, cost per real match, latency, with the unknown range for unlabeled picks.
`disagreements`	count + item ids	Items where A and B differ; the labeling queue.
`rule`	text	The pre-set switch rule, copied into the readout.
`decision`	switch \| keep \| shadow \| relabel	What happened next.

one replay run, as data

{
 "replay_id": "rp.judge.02",
 "component": "judge",
 "version_a": "judge_prompt_v1",
 "version_b": "judge_prompt_v2_examples",
 "dataset": {
 "source": [
 "reddit",
 "x_pain_search"
 ],
 "window": "2026-10-01/2026-10-16",
 "items": "all matches incl. discards",
 "frozen_snapshot": "snap_2026_10_16"
 },
 "golden_set": "gs.judge.01 (50 items)",
 "labels": {
 "source": "founder",
 "count": 200,
 "made_after_seeing_b": "flagged"
 },
 "metrics": null,
 "disagreements": null,
 "rule": "switch if P@20 worst case is +10 pts, recall not lower, cost per real match under the cap",
 "decision": null
}

the shadow verdict

Field	Type	What it holds
`item_id`	string	The raw item or match: one post, one listing, one review.
`lane_id`	string
`version`	version string	Which version produced this verdict.
`live`	boolean	True for the version that acted; false for shadow.
`verdict`	yes \| no + reason	The decision and its one-line reason.
`confidence`	0 to 1	As reported by the judge.
`cost`	dollars	Model and API cost for this verdict.
`latency_ms`	integer	Time to verdict, so slow versions show up before they ship.
`decided_at`	timestamp	When the verdict was made.

one item, two versions

[
 {
 "item_id": "t3_example",
 "version": "judge_prompt_v1",
 "live": true,
 "verdict": {
 "value": "yes",
 "reason": "asks how to find first customers for a live SaaS"
 },
 "confidence": 0.71,
 "cost": 0.0004,
 "latency_ms": 820,
 "decided_at": "2026-10-17T13:02:11Z"
 },
 {
 "item_id": "t3_example",
 "version": "judge_prompt_v2_examples",
 "live": false,
 "verdict": {
 "value": "no",
 "reason": "author is selling a service, not buying"
 },
 "confidence": 0.84,
 "cost": 0.0007,
 "latency_ms": 1140,
 "decided_at": "2026-10-17T13:02:12Z"
 }
]

catalog

Twelve changes you can test without touching a lead.

Each card names the dataset, the labels it needs, the metric that decides, and the guardrail. Anything here should be replayed before it's run live, and anything that reads a new source should shadow first.

stage 6 · Signal — Offline replay

Judge prompt v2

change: Add worked examples and a 'seller, not buyer' rule to the judge prompt.
data: matched posts with frozen text
labels: about 200 founder labels
metric: precision on the top 20 a day
guard: recall; cost per real match

stage 6 · Signal — Offline replay

Judge model swap

change: Distill the judge into a smaller, cheaper model.
data: the same matches the big judge graded
labels: the big judge's verdicts plus founder labels
metric: agreement with founder labels
guard: precision must not drop more than a few points

stage 5 · Detector — Shadow

Detector vocabulary v2

change: Add buyer phrases pulled from real replies; drop phrases that only match sellers.
data: the live feed for two weeks
labels: a labeled slice of new matches
metric: real matches per 100
guard: run cost per real match

stage 5 · Detector — Shadow

New source lane

change: A new source, such as the Reddit RSS replacement, beside the old one.
data: both sources over the same days
labels: labels on non-overlapping matches
metric: real matches per day; freshness
guard: duplicate rate; cost

stage 6 · Signal — Offline replay

Routing rules

change: Change which queue (pain, engage, listen, launch) a signal goes to.
data: routed signals with outcomes
labels: founder re-routes as labels
metric: share routed where the founder agreed
guard: no queue starved

stage 6 · Signal — Offline replay

Discard thresholds

change: Loosen or tighten the stale or excluded rules.
data: discards and their reasons
labels: a random weekly slice of discards, labeled
metric: real items recovered per 100 discards
guard: noise added to the list

stage 7 · Lead — Offline replay

Lead scorer weights

change: Weight signal freshness higher, or stacked signals.
data: past leads with frozen features
labels: replied within 7 days
metric: replies among the daily top 10
guard: no tier inflation

stage 7 · Lead — Offline replay

Tier cutoffs

change: Move the Hot and Warm lines.
data: scored leads
labels: founder accept, reply
metric: reply rate per tier, monotonic
guard: Hot list size the founder can work

stage 7 · Lead — Offline replay

Identity merge threshold

change: Merge on fuzzier matches across X, Reddit and the site.
data: identity pairs with method and confidence
labels: founder splits as negative labels
metric: merge precision
guard: merges per lead

stage 3 · ICP — Offline replay

Cut an ICP line

change: Drop a fit line that never predicted payment.
data: reviewed leads
labels: founder accept; paid
metric: agreement with accept and paid
guard: precision must not drop

stage 2 · Brief — Offline replay

Brief extractor v2

change: A second pass that quotes the source for every field.
data: stored product sites
labels: founder corrections per field
metric: fields kept unedited
guard: cost per brief

stage 4 · Trigger — Offline replay

Freshness window

change: Shorten a trigger's window from 72 hours to 24.
data: signals with age at first touch
labels: reply within 7 days
metric: reply rate by age bucket
guard: signals lost per day

failure modes

How offline tests lie, and how to catch it.

Tuning on the test set — Prompt v2 was written while looking at the same items it's graded on, so it memorized them. · Keep a golden set that no prompt author sees; it decides the switch.

Leaked features — A lead's later reply or an edited post made it into the inputs. · Inputs come only from frozen snapshots taken at decision time.

Only grading A's picks — No kept negatives, so B's new finds look like errors or don't count. · Report precision as a worst-to-best range; label disagreements to close it.

Drift — Last month's posts don't look like next month's, especially after a source change. · Always shadow after replay; compare shadow numbers to replay numbers before switching.

Labels shaped by the candidate — The founder labels items after seeing B's verdict and leans toward it. · Label blind to which version said what; flag labels made after seeing B.

Shadow costs double — Running two judges on every item doubles model spend. · Cap shadow to a sample of items, and log cost per verdict to see it.

Version not stamped — After a switch, later verdicts don't say which version made them. · Every verdict row carries its version; a switch is logged as a change.

Metric too coarse — Overall accuracy rises while the top of the daily list gets worse. · Decide on precision at the top 20, the part the founder actually works.

data to capture

What replay and shadow record, and what that trains.

Record — every verdict from every version with its version, confidence, cost and latency; shadow verdicts as rows with live false; each replay run with its dataset, labels, rule and decision; the golden set's ids

Label — founder real or not on disagreements and a random slice of agreements; founder re-routes and splits; outcomes (reply, paid) joined at grading time only

Trains — the match judge and its distilled small model; the scorer; the router; and a calibration of judge confidence against founder labels

Test — every listen-side change: replay first, then shadow, then switch with a version stamp

Backtesting and shadow deployment.

Replay first, then shadow, then switch.

Rule 3 (freeze the inputs) and rule 4 (keep the negatives).

Compare A and B on the same labeled items.

A replay run, and a shadow verdict row.

Twelve changes you can test without touching a lead.

Judge prompt v2

Judge model swap

Detector vocabulary v2

New source lane

Routing rules

Discard thresholds

Lead scorer weights

Tier cutoffs

Identity merge threshold

Cut an ICP line

Brief extractor v2

Freshness window

How offline tests lie, and how to catch it.

What replay and shadow record, and what that trains.

The components replay tests, and where results go.