phase 4 · learn
Tracking plan
Go deeper
What to record at every stage of the pipeline so that, months from now, the data can answer which triggers pay, which detectors are right, which leads reply, and which messages work, and can train models that do it automatically.
The one idea: log every decision the system makes, with what it knew at that moment, and later log what happened. A decision without its outcome teaches nothing; an outcome without the decision behind it can't be traced. Everything below is that idea applied stage by stage.
ten rules
How to record so the data stays usable.
Most data that "can't be used for a model" fails one of these. They cost little on day one and are close to impossible to add later.
01 · append-only — Never overwrite. Add events. · A lead's score changing is a new lead_scored event, not an update. You can always rebuild the current state from events; you can never rebuild history from the current state.
02 · one chain of ids — Every event carries the ids upstream of it. · A payment knows its touch, the touch knows its campaign and lead, the lead knows its signals, each signal knows its detector and trigger. That chain is attribution.
03 · point in time — Freeze the features at decision time. · When a lead is scored, store the inputs used, not just the score. Training on today's values for a decision made last month leaks the future and gives a model that only looks good on paper.
04 · version everything — Every rule and prompt gets a version. · Detector, judge prompt, model id, ICP, playbook, template. Without versions you can't tell whether replies rose because of the new opener or the new detector.
05 · keep the negatives — Log what you didn't do. · Skipped leads, rejected matches, drafts never sent, messages with no reply. A model learns as much from the no's, and they are the first thing people forget to store.
06 · log the odds — Record how a choice was made. · When the system picks one option from several (which lead to show, which opener to try), store every option and the probability each had. That's what makes fair comparisons possible later (propensity logging).
07 · keep a holdout — Don't contact everyone you could. · Leave a small random share of qualified leads alone. Comparing them with the contacted ones is the only clean answer to "did the message cause the signup, or would they have come anyway?"
08 · two clocks — When it happened and when you learned it. · occurred_at and recorded_at, both in UTC. A post from Tuesday caught on Thursday is two days stale, and freshness is one of the strongest features you'll have.
09 · keep the words — Store the raw text, not just a summary. · The matched post, the draft, the sent version, the reply. Text is what future models read; a sentiment score from today's judge is less useful than the sentence it judged.
10 · count the cost — Record what each step cost. · API spend per detector run, model tokens per draft, founder minutes per lead. "Which trigger pays" becomes "which trigger pays per dollar and per hour".
the chain of ids
Every payment traces back to the trigger that started it.
Each box is a record with its own id. Each arrow is an id the next record carries. Follow the arrows backwards from payment and you reach the post, the detector and the trigger behind the customer.
LISTEN · WHO IS READY
- trigger — trigger_id
- detector — detector_id + v
- run — run_id
- raw item — raw_item_id
- match — match_id
- signal — signal_id
- person — person_id
- lead — lead_id ACT AND OUTCOMES · WHAT HAPPENED
- assignment — campaign_id + arm
- touch — touch_id
- reply · click — touch_id
- visit — touch_id
- signup — first + last touch
- activation — account_id
- payment — Stripe invoice Read right to left to attribute a payment; read left to right to see what each trigger and detector produced.
the event envelope
Every event has the same outer shape.
One table of events, one shape. Names follow the object_action convention, past tense, so lead_scored, never score lead. The payload changes by event; the envelope never does.
event_id | Unique. Lets retries be written twice and counted once. |
event_type | From the catalog below. Nothing outside the catalog gets written. |
occurred_at | When it happened in the world (the post, the click, the payment). |
recorded_at | When the system learned of it. |
account_id | Whose pipeline. Every query starts here. |
actor | system, founder or lead. Who did it. |
refs | The id chain: trigger, detector, signal, person, lead, campaign, touch. Only the ones that exist yet. |
versions | Every rule or prompt that shaped this event, with its version. |
payload | The event's own fields, including the features used if it's a decision. |
cost | Optional: dollars, tokens, founder seconds. |
// one lead_scored event
{
"event_id": "evt_01JAX…",
"event_type": "lead_scored",
"occurred_at": "2026-10-16T14:02:11Z",
"recorded_at": "2026-10-16T14:02:11Z",
"actor": "system",
"refs": {
"lead_id": "lead_jane",
"person_id": "per_7f3",
"signal_ids": ["sig_101","sig_102","sig_109"]
},
"versions": {
"scorer": "v3", "icp": "v2"
},
"payload": {
"score": 0.81, "tier": "hot",
"features": {
"triggers": 3, "freshest_hours": 9,
"warm": true, "icp_lines_matched": 4
}
},
"cost": { "usd": 0.0004 }
}event catalog
What gets logged at each stage.
Grouped by pipeline stage. Events marked label are ground truth: the answers a model learns from. Everything else is a decision or a feature.
understand — Brief · stage 2
| Event | When it fires | Key fields |
|---|---|---|
brief_versionedby system · founder | the brief is drafted or edited | brief_version, every field with its source, evidence level (A to D) and date, product stage |
brief_confirmedby founder | the founder signs off | brief_version, fields the founder corrected and how label · which automatic fields were wrong |
competitor_mappedby system | research adds or updates a competitor | name, kind (direct, alternative, do-it-yourself), channels worked, price |
understand — ICP · stage 3
| Event | When it fires | Key fields |
|---|---|---|
icp_versionedby system · founder | the filter is written or changed | icp_version, each fits-if and not-if line with its evidence level, test results (list, count, vouch) |
understand — Triggers · stage 4
| Event | When it fires | Key fields |
|---|---|---|
trigger_judgedby system · founder | each of the 24 triggers is screened | trigger, status (start, on, later, off), the reason, icp_version |
listen — Detectors · stage 5
| Event | When it fires | Key fields |
|---|---|---|
detector_versionedby founder · system | a detector is created or its rule changes | detector_id, version, trigger, source, kind, match rule, strength, freshness window, exclusions |
detector_run_completedby system | each scheduled run finishes | run_id, items scanned, matches, errors, duration, API cost |
listen — Signals · stage 6
| Event | When it fires | Key fields |
|---|---|---|
item_matchedby system | a raw post, listing or record matches a rule | raw_item_id, url, author handle, posted time, full text, which rule fired, rule score |
match_judgedby system | the judge decides if it's really the trigger | verdict, reason, confidence, judge_model, judge_prompt_version |
match_labeledby founder | the founder marks a match real or not | correct, wrong, unsure, and why if given label · is this really the trigger |
signal_createdby system | a match passes freshness, exclusions and the judge | signal_id, trigger, strength, age in hours at capture |
match_discardedby system | a match fails a check | which check failed (stale, excluded, judge said no, no person) |
listen — Leads · stage 7
| Event | When it fires | Key fields |
|---|---|---|
identity_linkedby system | two records are judged to be one person | person_id, the handles, domains or emails joined, the method, confidence |
identity_unlinkedby founder | the founder splits a wrong merge | which link was wrong label · was the merge right |
lead_scoredby system | a lead's score is computed or changes | score, tier, scorer_version, and the features used, frozen |
lead_reviewedby founder | the founder acts on a lead | accepted, skipped, snoozed, never contact, reason label · does this person fit |
act — Playbooks and campaigns · stages 8 and 9
| Event | When it fires | Key fields |
|---|---|---|
campaign_startedby founder | a campaign begins | playbook id and version, channel, offer, segment rule, window, stop rule, arms and their shares |
lead_assignedby system | a lead enters a campaign | campaign_id, arm (treatment, holdout, variant), assignment_probability |
campaign_stoppedby system · founder | the window ends or the stop rule fires | why it stopped, final counts per step |
act — Sequence · stage 10
| Event | When it fires | Key fields |
|---|---|---|
draft_generatedby system | a message is drafted | touch_id, step, channel, template id, model, prompt_version, full text, token cost |
draft_editedby founder | the founder changes the draft | final text, edit distance, which parts changed label · how good the draft was |
draft_discardedby founder | the founder throws a draft away | reason if given label · how bad the draft was |
touch_sentby founder | the founder confirms it went out | touch_id, channel, sent time, final text |
post_publishedby system · founder | a public post goes out | post id, channel, text, scheduled versus actual time |
outcomes — Responses · stage 11
| Event | When it fires | Key fields |
|---|---|---|
reply_receivedby lead | the person answers | touch_id, full text, received time, class (positive, question, not now, stop) label · did the message work |
link_clickedby lead | a tracked link is opened | touch_id, link id, time label · did they act on it |
negative_feedbackby lead | they say stop, block or report | which, and on which touch label · never repeat this |
visit_startedby lead | they land on the site | touch_id from the link, landing page, referrer |
outcomes — Conversion · stage 11
| Event | When it fires | Key fields |
|---|---|---|
signup_createdby lead | an account is made | account id, first and last touch_id, the answer to "how did you hear about us?" label · did they sign up |
activation_reachedby lead | they do the first action that predicts paying | which action, time since signup label · did they activate |
payment_succeededby system | Stripe reports a paid invoice (<code>invoice.paid</code>) | amount, plan, customer id, first or renewal label · did they pay |
payment_failedby system | Stripe reports a failed payment (<code>invoice.payment_failed</code>) | amount, attempt number label · warm trigger: payment failed |
subscription_canceledby system | Stripe reports a cancellation (<code>customer.subscription.deleted</code>) | plan, tenure, the reason given label · did they churn |
learn — Loop · stage 12
| Event | When it fires | Key fields |
|---|---|---|
snapshot_takenby system | the daily snapshot runs | counts per step, new signals, new leads, alerts raised |
review_completedby founder | the founder reviews the 7-day window | counts per step, sample sizes, the constraint chosen, the hypothesis |
change_shippedby founder | one thing changes | what changed, where, the new version string, the date to judge it |
detector_retiredby founder | a detector or playbook is switched off | why, and its record: signals, leads, replies, payments |
where the answers come from
Three kinds of labels.
A model needs examples with the right answer attached. In this pipeline the answers arrive three ways, at very different speeds.
fast · minutes
Founder judgments
Every time the founder accepts, skips, edits or rejects something, that's a label.
- this match is real / not the trigger
- this lead fits / never contact
- sent the draft as is / edited it / threw it away
medium · days
Lead behavior
What the person did after a touch. Each needs a fixed window, decided up front, or late answers get counted as no's.
- replied within 7 days
- clicked within 7 days
- signed up within 14 days
- said stop, blocked or reported
slow · weeks
Money
The label that matters most and arrives last. Only Stripe writes it.
- paid within 30 days of first touch
- still paying at 90 days
- expanded, downgraded or cancelled
1 · listen — Match judge · Is this post really the trigger? The first model to build, because founder labels arrive fastest.
- label
match_labeled- features
- the post text, the source, which rule fired, the author's bio
- measure
- precision at the top of the daily list
- ready when
- 200 labeled matches to evaluate; about 1,000 to fine-tune
2 · listen — Detector precision · Which detectors produce real signals and which produce noise, per source and per rule.
- label
match_labeled, signal_created- features
- detector version, source, rule, time of day
- measure
- share of matches labeled real, per detector
- ready when
- a few weeks of labels per detector
3 · listen — ICP fit · Does this person fit the filter? Learns the founder's taste and what paying customers had in common.
- label
lead_reviewed, payment_succeeded- features
- bio, site, company size, which ICP lines matched
- measure
- agreement with the founder's accept or never
- ready when
- 100 reviewed leads
4 · act — Reply propensity (lead score v2) · Who is most likely to answer? Replaces the hand-written score once there's enough data.
- label
reply_received within 7 days- features
- the frozen features from lead_scored: triggers, freshness, warm or cold, fit lines
- measure
- replies among the top 10 a day
- ready when
- several hundred touches; sooner when pooled
5 · act — Opener and playbook choice · Which opener or playbook to use for this lead, learned from past randomized assignments.
- label
reply_received, signup_created- features
- trigger, channel, template, length, timing, logged assignment probability
- measure
- replies per touch versus the current default
- ready when
- randomized variants with logged probabilities
6 · act — Draft quality · How close is the draft to what the founder actually sends? An eval for every prompt change.
- label
draft_edited, draft_discarded- features
- the draft, the final text, the template, the prompt version
- measure
- share sent unedited; median edit distance
- ready when
- every draft is a case from day one
7 · outcome — Conversion and churn · Who will pay, who will cancel. Built on the warm triggers.
- label
payment_succeeded, subscription_canceled- features
- usage trend, activation, plan, tenure, touches received
- measure
- calibrated probability per account
- ready when
- months of payments; pooled across accounts
clean comparisons
Experiments need to be designed in, not added later.
Holdout group
Randomly keep about 10% of qualified leads out of every campaign, and log them as lead_assigned with arm: "holdout". Then track their signups exactly like everyone else's.
- What it answers: how many signups the messages caused, versus how many would have happened anyway.
- The cost: a few people a month you don't contact. At small volume, read it as a direction over months, not a weekly number.
Randomized variants
When two openers, offers or send times compete, assign them at random and log the probability each lead had of getting each one (assignment_probability).
- What it answers: which variant gets more replies, without the bias of always giving the better-looking leads the newer opener.
- Later: those logged probabilities let a model learn the best choice per lead from past data (a contextual bandit), instead of running a new test every time.
guardrails
What not to do with this data.
Public and own data only — Signals come from public posts and listings; warm events from your own product and Stripe. Nothing scraped from behind a login, nothing bought about individuals.
Delete on request — One person id ties together everything about someone, so a "remove me" request deletes all of it in one pass. Log the deletion itself, without the data.
Retention limits — Raw text about people who never became leads expires on a schedule (for example 12 months). Counts and labels can stay; the words go.
No leakage — A model may only use what was known before the decision it's predicting. Check every feature against occurred_at, every time.
Pooled learning needs consent — Training one model on many customers' data is where the real signal is, and it needs clear terms, opt-in or anonymization. Messages and replies never leave the account they belong to.
Feedback loops — A model that picks who to contact only sees outcomes for people it picked. The holdout and the logged probabilities are what keep it from confirming its own guesses.