learn · tracking plan
Labels
Human decisions and outcomes as ground truth.
In Tracking plan
Human decisions and real outcomes as ground truth.
rule 1 — Capture in the moment · A label asked for later is a guess.
rule 2 — Wait out the window · An open window is not a no.
rule 3 — Join to frozen features · Never to what is known today.
what it is, in industry terms
A label is the answer a model is trained to predict.
Industry names: ground truth, supervision signal, outcome label. Three kinds feed the pipeline: what the founder decided (judgment), what the person did (outcome), and what a randomized change caused (experiment). Only the third tells you cause.
Judgment label — human annotation · The founder marks a match real, skips a lead, edits a draft. Fast and plentiful; reflects taste.
Outcome label — observed outcome · A reply, a sign-up, a payment. Slow and sparse; reflects reality.
Experiment label — counterfactual outcome · The outcome under a randomly assigned arm. The only label that measures what a change caused.
Label window — observation window · How long to wait before "no" means no. A touch with no reply after 2 days is not yet a no.
Censoring — right-censoring · A unit whose window hasn't closed. Leave it out of training, or model the time.
Proxy label — surrogate · An easier signal standing in for the real one, like likes for interest. Check it predicts the real one.
how it works
From a decision or outcome to a training row.
- Event happens — decision or outcome
- Label attached — source, time, labeler
- Window closes — no means no
- Joined to features — frozen at decision
- Training row — dataset version
- Event happens — decision or outcome
- Label attached — source, time, labeler
- Window closes — no means no
- Joined to features — frozen at decision
- Training row — dataset version
Labels only join to features frozen at the moment of the decision, never to what is known today.
Capture the decision in the moment — One tap in the daily list: real or not. A label asked for a week later is a guess.
Record who labeled and how sure — Founder, judge model, or outcome. Unsure is a valid answer and is kept.
Wait out the window — Reply labels close at 7 days, sign-up labels at 14 to 30. Until then the row is censored.
Join to frozen features — Use the feature snapshot written when the decision was made (lead_scored, draft generated), not today's values.
Version the dataset — A training set is a query plus a date. Store both so a model can be retrained on the same rows.
the object
One label record.
label
| Field | Type | What it holds |
|---|---|---|
label_id | uuid | |
subject | type + id | What is labeled: match, lead, draft, touch, account. |
label_type | string | One of the catalog entries. |
value | string or number | The answer. |
source | founder | outcome | judge | experiment | Where it came from. |
labeler | string | Who or what gave it. |
confidence | number or null | For judge labels and "unsure". |
observed_at | timestamp | When the answer became known. |
window | duration | How long it was waited for. |
censored | boolean | Window not yet closed. |
features_ref | event id | The frozen feature snapshot it pairs with. |
note | text | The founder's reason, verbatim. |
{
"label_id": "lab_31…",
"subject": {
"type": "draft",
"id": "draft_8f2…"
},
"label_type": "draft decision",
"value": "shipped_edited",
"source": "founder",
"observed_at": "2026-10-17T14:02:11Z",
"window": "at decision",
"censored": false,
"features_ref": "draft.generated:draft_8f2…",
"note": null
}the catalog
Every label the pipeline produces.
human judgment · the founder's decisions — 7
judgment
match real or not
Given by founder, on a matched post.
- values
- real · wrong · unsure
- arrives
- minutes
- trains
- match judge, detector precision
judgment
lead accept, skip, never
Given by founder, on a lead.
- values
- accept · skip · snooze · never
- arrives
- hours
- trains
- ICP fit
judgment
identity split
Given by founder, on a merge.
- values
- right · wrong
- arrives
- days
- trains
- entity resolution
judgment
brief field corrected
Given by founder, on one brief field.
- values
- kept · corrected · rejected
- arrives
- at sign-off
- trains
- brief extractor evals
judgment
draft decision
Given by founder, on a draft.
- values
- shipped · shipped edited · discarded
- arrives
- seconds to hours
- trains
- draft quality, prompt evals
judgment
edit locus
Given by founder, on an edited draft.
- values
- hook · body · ending · rewrite
- arrives
- at decision
- trains
- which part of the prompt to fix
judgment
preference pair
Given by founder, on draft vs posted text.
- values
- posted text preferred
- arrives
- at decision
- trains
- fine-tuning the drafter
outcomes · what really happened — 9
outcome
reply within 7 days
Given by lead, on a touch.
- values
- yes · no (censored until day 7)
- arrives
- 7 days
- trains
- reply propensity
outcome
reply class
Given by lead · judge, on a reply.
- values
- positive · question · not now · stop
- arrives
- hours
- trains
- reply classifier, opener choice
outcome
clicked
Given by lead, on a touch or post.
- values
- yes · no
- arrives
- 7 days
- trains
- link CTR models
outcome
signed up / joined waitlist
Given by lead, on a lead or visit.
- values
- yes · no
- arrives
- 14 to 30 days
- trains
- conversion
outcome
activated
Given by lead, on an account.
- values
- yes · no
- arrives
- 7 days
- trains
- activation model
outcome
paid
Given by Stripe, on an account.
- values
- yes · no, amount
- arrives
- trial length
- trains
- conversion and LTV
outcome
churned
Given by Stripe, on a subscription.
- values
- canceled, tenure, reason
- arrives
- months
- trains
- churn
outcome
stop or report
Given by lead, on a touch.
- values
- stop · block · report
- arrives
- any time
- trains
- guardrail, contact policy
outcome
post performance
Given by the channel, on a post.
- values
- impressions, profile clicks at 5 checkpoints
- arrives
- 7 days
- trains
- early forecast, send time
experiments · what a change caused — 2
experiment
arm outcome
Given by randomization, on a unit in a test.
- values
- converted under arm A or B
- arrives
- test window
- trains
- causal effect, uplift
experiment
experiment verdict
Given by readout, on a change.
- values
- win · lose · inconclusive
- arrives
- judge-by date
- trains
- priors for the next test
failure modes
How labels mislead.
| Failure | Symptom | Catch | Fix |
|---|---|---|---|
| Leakage | great offline accuracy, poor live | features computed after the label | only join to frozen snapshots |
| Selection bias | the model only knows people you chose | labels exist only for touched leads | a small random exploration slice |
| Early "no" | reply rates look low | labels read before the window closed | treat open windows as censored |
| Taste drift | old labels disagree with new ones | relabel a sample monthly | weight recent labels; note why |
| Proxy mistaken for goal | likes go up, joins don't | correlation of proxy to real outcome | report both; optimize the real one |
| Inconsistent labeler | the same post labeled both ways | repeat a few items blind | write the labeling rule down |
data to capture
What this component records, and what that data trains.
record
- every label with source, labeler, time, window and censoring
- the founder's reason, verbatim
- unsure answers, kept
label
- this page is the label list
trains
- match judge, ICP fit, reply propensity, draft quality, conversion, churn, uplift
test
- label coverage per week
- a blind relabel of 20 items a month for consistency