learn · tracking plan
Models
The seven models and when each is ready.
In Tracking plan
The models the pipeline can grow into, what each predicts, the label it needs, when there is enough data, and the hand-written baseline it has to beat first.
rule 1 — Beat the baseline twice · Offline on frozen data, then live on a randomized slice.
rule 2 — Wait for the labels · "Ready when" is a number of labels, not a date.
rule 3 — Keep exploring · A random slice stops the model learning only from itself.
what it is, in industry terms
Each model replaces one hand-written rule, once the labels are there.
Industry names: model registry, baseline, champion and challenger. Every model starts as a rule or a prompt (the baseline). It is replaced only when a trained model beats it on frozen data and then again in a live test.
Baseline — heuristic · The rule or prompt in use today. Every model must beat it.
Champion — production model · The version currently deciding.
Challenger — candidate · A new version run in shadow or on a slice.
Offline eval — backtest · Scored on frozen, labeled history.
Shadow — dark launch · Decides alongside the champion without acting.
Drift — distribution shift · Inputs or outcomes change over time and the model gets worse.
how it works
The lifecycle every model follows.
- Baseline rule — in use today
- Collect labels — until ready
- Offline eval — frozen data
- Shadow — decides, doesn't act
- Live slice — randomized test
- Promote — new champion
- Monitor — drift and guardrails
- Baseline rule — in use today
- Collect labels — until ready
- Offline eval — frozen data
- Shadow — decides, doesn't act
- Live slice — randomized test
- Promote — new champion
- Monitor — drift and guardrails
A model ships only after beating the baseline twice: offline on frozen data, then live on a randomized slice.
Name the decision — One question, one label, one owner.
Write the baseline down — The current rule, versioned, so there is something to beat.
Wait for "ready when" — Too few labels gives a model that memorizes.
Evaluate on frozen data — Time-split: train on older, test on newer.
Shadow, then a live slice — Through the experimentation engine, with a pre-set win rule.
Keep exploring — A small random slice keeps the labels honest, so the model isn't trained only on its own choices.
the object
One model registry entry.
model
| Field | Type | What it holds |
|---|---|---|
model_id | string | m.<name> |
version | string | |
decision | sentence | What it decides. |
label | label type | From the label catalog. |
features | list + hash | Feature names; the hash detects silent changes. |
training_window | dates | Which rows it learned from. |
dataset_ref | query + date | Reproducible training set. |
baseline | string | What it must beat. |
offline_metrics | object | Scores on the holdout period. |
status | shadow | slice | champion | retired | |
experiment_id | string | The live test that promoted it. |
{
"model_id": "m.match_judge",
"version": "v2-candidate",
"decision": "is this post really the trigger",
"label": "match real or not",
"features": [
"text",
"source",
"rule",
"author_bio"
],
"training_window": [
"2026-11-01",
"2026-12-15"
],
"baseline": "lane judge prompt v4",
"offline_metrics": {
"precision_at_10": "to measure"
},
"status": "shadow",
"experiment_id": null
}the catalog
Every model the pipeline can grow into.
Seven from the tracking plan, plus entity resolution, send time and the early post forecast.
listen · who is ready — 4
listen
Match judge
Is this post really the trigger?
- label
- match real or not
- features
- post text, source, rule fired, author bio
- measure
- precision at the top of the daily list
- ready when
- 200 labels to evaluate, about 1,000 to fine-tune
- baseline
- prompt-based judge per lane
listen
Detector precision
Which detectors produce real signals?
- label
- match real or not, signal created
- features
- detector version, source, rule, time
- measure
- share of matches real, per detector
- ready when
- a few weeks of labels per detector
- baseline
- counts per detector
listen
ICP fit
Does this person fit the filter?
- label
- lead accept or never, paid
- features
- bio, site, size, ICP lines matched
- measure
- agreement with the founder
- ready when
- 100 reviewed leads
- baseline
- the public-pain rule
listen
Entity resolution
Are these two records one person?
- label
- identity split
- features
- handles, domains, bios, links, timing
- measure
- merge precision
- ready when
- a few hundred reviewed merges
- baseline
- deterministic keys
act · what to send — 4
act
Reply propensity (lead score v2)
Who is most likely to answer?
- label
- reply within 7 days
- features
- frozen features from lead scoring
- measure
- replies among the daily top 10
- ready when
- several hundred touches
- baseline
- hand-written tiers
act
Opener and playbook choice
Which opener or play for this lead?
- label
- reply, sign-up
- features
- trigger, channel, template, timing, logged propensity
- measure
- replies per touch vs default
- ready when
- randomized variants with logged odds
- baseline
- the next-best-action ranking
act
Draft quality
How close is the draft to what ships?
- label
- draft decision, edit distance
- features
- draft, final text, template, prompt version
- measure
- share shipped unedited; median edit distance
- ready when
- every draft, from day one
- baseline
- prompt versions with randomized slots
act
Send time
When will this post or message do best?
- label
- post performance at 24 hours
- features
- hour, day, account, format
- measure
- lift over best fixed hour
- ready when
- hundreds of posts per account
- baseline
- engagement by day and hour
outcomes · what happens next — 2
outcome
Early post forecast
Will this post land?
- label
- impressions at 7 days
- features
- the 10 and 30 minute snapshots
- measure
- forecast error
- ready when
- already enough posts
- baseline
- snapshots exist
outcome
Conversion and churn
Who will pay, who will cancel?
- label
- paid, canceled
- features
- usage, activation, plan, tenure, touches
- measure
- calibration
- ready when
- months of payments, pooled
- baseline
- Stripe webhooks
illustrative example
The first two models to build, and why.
Draft quality first. Its labels come with every draft, with randomized slots and prompt versions. A reward model trained on (draft, posted text) pairs can score new prompt versions offline before they reach the founder.
Match judge second. A prompt-based judge usually runs first. Once one-tap real or not labels exist, 200 of them are enough to measure the prompt's precision, and about 1,000 to try a cheaper distilled judge in shadow.
Reply propensity and conversion wait on touch_id and on enough sent messages.
failure modes
How models fail in a small pipeline.
| Failure | Symptom | Catch | Fix |
|---|---|---|---|
| Too few labels | perfect offline, random live | learning curve flat or noisy | wait; keep the baseline |
| Feedback loop | the model only sees people it picked | no labels outside its picks | random exploration slice |
| Leakage | too good to be true | features after decision time | frozen snapshots only |
| Drift | quality slides month to month | monthly re-score on fresh labels | retrain on a rolling window |
| Silent feature change | scores shift overnight | feature hash differs from registry | block promotion on hash mismatch |
| Optimizing a proxy | more replies, fewer joins | track the downstream guardrail | promote on the real outcome |
data to capture
What this component records, and what that data trains.
record
- registry entry per version
- every prediction with model version and score
- the exploration flag on every decision
label
- models consume labels; they don't create them
trains
- their own successors: each champion's predictions become features for the next challenger
test
- offline eval on a time-split
- shadow agreement rate
- live slice through the experimentation engine