learn · improvement loop
Threshold catalog
Every metric's trigger level, per stage.
In Improvement loop
Every line the pipeline watches, stage by stage: the metric, the kind of threshold, a starting level, the minimum sample before it may fire, how loud it is, and what it does when it fires. Levels are starting rules of thumb, not sourced numbers. The catalog exists so they can be re-fitted from real firings.
metric → threshold · level, window, gate → firing → action → label
what it is, in industry terms
An alerting rulebook, written as data.
Monitoring tools call these alert rules; reliability teams call the reputation ones guardrails and the quality ones SLOs (service level objectives). Here every rule is a row with a version, so the catalog can be filtered, audited and re-fitted, and a firing always points back to the exact rule that fired. Seven kinds cover everything the pipeline needs.
Floor — A rate must stay above a level: detector precision, play signups.
Ceiling — A rate must stay below a level: stop requests, correction rate, never rate.
Control limit — Fires when a point falls outside limits drawn from the metric's own history: reply rate per play, rung conversion.
Change vs baseline — Fires on a relative move against a rolling baseline: cost per lead up 2×, edit distance up a third.
Staleness — Fires when something hasn't happened for too long: source freshness, Hot lead untouched, overdue readout.
Sample-size gate — Not a line by itself: the minimum units before any other line may fire. Every row has one.
Event — A single event that always matters: Stripe invoice.payment_failed starts a play.
how a line fires
Five checks, in order, before anything happens.
- Read metric — over its window from the snapshot
- Sample gate — enough units? else stay silent
- Compare — value vs level or control limits
- Cooldown — and mute fired recently? muted?
- Fire — write alert row run action
Cooldown stops the same line from firing on the same scope again before its gap has passed, e.g. 7 days for a detector precision line. Muted lines still log, so false-alarm counts stay honest.
the fields, as the catalog uses them
Read a row left to right.
The full threshold object is on the Improvement loop page. These are the columns below and how to set each one.
| Column | Set it by | Watch out for |
|---|---|---|
| Stage · metric | Point at one metric definition on Funnel and metrics; never write the formula into the threshold. | Two thresholds computing the same metric two ways will disagree and both be right. |
| Kind | Pick by the metric: rates get control limits once there's history; reputation gets ceilings; time gets staleness. | A fixed floor on a rate with tiny samples fires on noise. |
| Level | Start from the rule of thumb or the baseline's control limit, whichever is wider. Re-fit monthly. | Copying a level from another account. Baselines differ by channel and audience. |
| Window | Long enough to hold the minimum sample at normal volume. | A 24-hour window on a metric that sees 5 units a day. |
| Min sample | The smallest count where a crossing means something. For rates, at least 30 units is a sane floor. | Leaving it at zero: the first unlucky day fires. |
| Severity | critical pages and may auto-pause; warn notifies; info logs. | Everything critical. Alert fatigue arrives in a week. |
| Action · owner | The least-reaching action that fixes it. System may pause, shadow, alert, log; the agent may replay and propose; the founder approves anything live. | An auto-action that changes what a lead sees without approval. |
the catalog
Every threshold, all twelve stages.
Levels are starting rules of thumb. Stars mark the twenty starter lines on the Improvement loop page.
| Stage · metric | Kind | Level (rule of thumb) · window | Min sample | Severity | Action · owner | Why |
|---|---|---|---|---|---|---|
| ★ stage 1 · Link Intake completion rate (links in → finished briefs) thr.link.intake_drop | Change vs baseline | Down 30% against its 4-week baseline window: 7 days | 30 intake starts | warn | Open hypothesis founder | A copy or step change broke the intake. Opens an intake copy or step-order test. |
| stage 1 · Link Intake step error rate thr.link.step_errors | Ceiling | Over 5% of runs error on any one step window: 24 hours | 20 runs | critical | Alert only system | Something is broken, not underperforming. Alert first; nothing to test until it's fixed. |
| ★ stage 2 · Brief Founder correction rate, per brief field thr.brief.field_corrections | Ceiling | Over 25% of one field's values corrected window: 30 days | 20 briefs | warn | Queue offline replay agent | The extractor is wrong on that field. Replay the next prompt version on every corrected case before shipping it. |
| stage 2 · Brief Share of tier A facts the founder corrected thr.brief.tier_a_wrong | Ceiling | Over 10% window: 30 days | 30 tier A facts | warn | Open hypothesis agent | Tier A is supposed to mean quoted and checked. If it's wrong this often, the evidence rules need a test. |
| stage 2 · Brief Cost per brief run thr.brief.cost | Change vs baseline | Up 50% against its 4-week baseline (a cost rise is the bad direction) window: 7 days | 10 runs | info | Alert only system | Usually a crawl or a prompt grew. Alert only. |
| ★ stage 3 · ICP Share of reviewed leads the founder marks never thr.icp.never_rate | Ceiling | Over 40% window: 7 days | 30 reviews | warn | Open hypothesis agent | The filter lets in people the founder won't contact. Test tighter not-if lines by replaying past reviews. |
| stage 3 · ICP Lift of one fit line on reply or payment thr.icp.dead_line | Floor | Leads matching the line reply or pay no more than leads without it window: 90 days | 100 leads on each side | info | Propose retire founder | A fit line that predicts nothing is noise. Propose retiring it; the founder approves. |
| ★ stage 4 · Trigger Reply rate on signals older than the trigger's freshness window thr.trig.decay | Floor | Under half the reply rate of fresh signals window: 60 days | 40 touches in each age bucket | info | Propose test agent | The window is too long. Propose a shorter fresh_for, judged on the next 60 days. |
| stage 4 · Trigger Signals from a live trigger thr.trig.silent | Staleness | None in 14 days window: 14 days | n/a | warn | Alert only system | A live trigger that never fires is either rare or broken. Check its detectors' health. |
| ★ stage 5 · Detector Detector precision (founder labels real ÷ labeled) thr.det.precision | Floor | Under 60% over the last 50 founder labels window: rolling 50 labels | 50 labels | warn | Auto-shadow system | Move the detector to shadow so it stops filling the list, and open a rule test. |
| stage 5 · Detector Cost per accepted lead, per detector thr.det.yield_cost | Change vs baseline | Up 2× against its 4-week baseline window: 30 days | 20 accepted leads | warn | Open hypothesis agent | Same leads for more money: vocabulary is too broad or a source got noisy. |
| ★ stage 5 · Detector Matches per day, per detector thr.det.spike | Control limit | Above the upper control limit (about 3× the 4-week daily median as a first cut) window: daily | 14 days of history | warn | Alert only system | Spam waves and viral threads look like demand. Alert; shadow it if precision drops too. |
| ★ stage 5 · Detector Source freshness (newest item age) thr.src.stale | Staleness | Past the source's own max staleness window: continuous | n/a | warn | Alert only system | Mark the source's health stalled and alert. |
| ★ stage 5 · Detector Share of runs ending partial or error, per source thr.src.errors | Ceiling | Over 10% of runs in 24 hours window: 24 hours | 10 runs | warn | Alert only system | |
| ★ stage 6 · Signal Judge and founder disagreement on labeled signals thr.sig.disagree | Ceiling | Over 20% window: 14 days | 40 labels | warn | Queue offline replay agent | Replay the next judge prompt on every labeled case; ship only if agreement goes up and recall doesn't drop. |
| stage 6 · Signal Share of matches discarded as no person thr.sig.noperson | Change vs baseline | Over 2× its 4-week baseline window: 7 days | 100 matches | info | Alert only system | Usually the identity step broke, or a source changed its author field. |
| ★ stage 6 · Signal Oldest untouched item in the pain queue thr.sig.queue_age | Staleness | Over 72 hours window: continuous | n/a | warn | Alert only founder | Signals decay. Tell the founder before the window closes. |
| ★ stage 7 · Lead Reply rate of the daily top 10 thr.lead.top10 | Control limit | Below the lower control limit for 2 weeks running window: weekly | 30 touches a week | warn | Propose test agent | The ranking stopped finding responders. Propose a scorer change, replayed on frozen features first. |
| stage 7 · Lead Share of merges the founder splits thr.lead.splits | Ceiling | Over 5% window: 30 days | 40 merges | warn | Queue offline replay agent | The identity rules are merging strangers. Tighten and replay. |
| ★ stage 7 · Lead Hot lead with no touch thr.lead.hot_aging | Staleness | Over 48 hours window: continuous | n/a | warn | Alert only founder | A hot lead going cold is the most expensive miss in the pipeline. |
| ★ stage 8 · Playbook Reply rate per touch, per play thr.pb.reply | Control limit | Below the play's lower control limit window: rolling 4 weeks | 30 touches | warn | Propose test agent | Open an opener test on that play. Don't change the play and the opener at once. |
| stage 8 · Playbook Share of moves taken from the exploration slice thr.pb.explore | Floor | Under 5% of moves window: 7 days | 100 moves | info | Alert only system | Without exploration, the logged odds stop supporting offline evaluation. Data-quality alert. |
| stage 8 · Playbook Signups from a play thr.pb.dead_play | Floor | Zero after 60 touches window: since launch | 60 touches | info | Propose retire founder | Propose retiring or reworking the play. The founder decides. |
| ★ stage 9 · Campaign Stop requests per 100 touches thr.camp.stops | Ceiling | Over 2 per 100 window: 7 days | 50 touches | critical | Auto-pause system | Pause the campaign now. Reputation damage compounds faster than any test can measure. |
| ★ stage 9 · Campaign Holdout leads who received a touch thr.camp.holdout | Ceiling | Any window: continuous | n/a | critical | Auto-pause system | A touched holdout voids the lift readout. Pause and alert. |
| stage 9 · Campaign Campaign past its readout date with no readout thr.camp.overdue | Staleness | Over 48 hours late window: continuous | n/a | info | Alert only founder | Unread tests teach nothing. Nudge the readout. |
| ★ stage 10 · Sequence Median draft edit distance thr.seq.edit_drift | Change vs baseline | Up by a third week over week window: weekly | 20 decided drafts | warn | Queue offline replay agent | The drafter drifted from the founder's voice. Replay the current prompt against recent edits to find where. |
| stage 10 · Sequence Share of drafts shipped unedited thr.seq.unedited | Change vs baseline | Down 25% against its 4-week baseline window: weekly | 20 decided drafts | warn | Propose test agent | Same signal as edit drift, from the other side. Propose a prompt test. |
| stage 10 · Sequence Scheduled sends that missed their grace window thr.seq.missed | Ceiling | Over 2% of due sends window: 24 hours | 20 due sends | warn | Alert only system | A dispatcher or token problem, not a content problem. |
★ stage 11 · Funnel thr.fun.pace | Floor | Behind the line window: weekly | n/a | warn | Monday review founder | The Monday review picks the weakest rung as the constraint and opens one hypothesis on it. |
| ★ stage 11 · Funnel Conversion of one funnel rung thr.fun.rung | Control limit | Below its lower control limit 2 weeks running window: weekly | 30 entering the rung | warn | Open hypothesis agent | That rung is the constraint. Open one hypothesis on it, not three. |
| stage 11 · Funnel Stripe invoice.payment_failed thr.fun.payfail | Event | Any window: event | n/a | warn | Start a play system | Not a threshold so much as a warm trigger: start the failed-payment recovery sequence. |
| stage 11 · Funnel A post's 30-minute impressions thr.fun.early | Floor | Below the account's 10th percentile at 30 minutes window: per post | the account's post history | info | Log only system | Logged for the early-read forecast. No action: a single post is noise. |
| ★ stage 12 · Loop Experiment past its judge-by date with no readout thr.loop.overdue | Staleness | Over 48 hours window: continuous | n/a | warn | Alert only founder | Every test gets read, win or lose. |
| ★ stage 12 · Loop Share of a threshold's firings labeled not real thr.loop.falsealarm | Ceiling | Over 50% of its last 10 firings window: last 10 firings | 10 firings | info | Re-fit threshold agent | The threshold cries wolf. Re-fit its level from the last 8 weeks and version the change. |
| stage 12 · Loop Founder minutes spent on the pipeline thr.loop.minutes | Ceiling | Over the week's stated budget window: weekly | n/a | info | Alert only founder | A guardrail on the founder's time: the pipeline should save hours, not eat them. |
illustrative example
The first eight lines to switch on, before launch.
Everything else runs silent (log only) until it has a baseline.
Stop requests per 100 touches — Stop requests over 2 per 100 touches auto-pauses a campaign. The one line that protects the founder's name.
Holdout leads who received a touch — Any touched holdout pauses the campaign. Without it, every lift readout is quietly void.
Source freshness (newest item age) — A stale source alerts and is marked stalled. Reddit RSS ends Nov 13, 2026: this is the line that notices.
Detector precision (founder labels real ÷ labeled) — Under 60% over the last 50 labels shadows the detector. Needs labels first, so it switches on once 50 exist.
Hot lead with no touch — A Hot lead untouched for 48 hours alerts the founder. The cheapest win on the list.
Oldest untouched item in the pain queue — The pain queue's oldest item past 72 hours alerts. Same idea one step earlier.
Experiment past its judge-by date with no readout — A test past its judge-by date with no readout alerts. Keeps the loop honest from day one.
failure modes
How a catalog goes wrong.
Alert fatigue — Too many lines at warn or critical; the founder stops reading them within a week. · Count firings per week per severity. Over a handful of non-critical firings a week means demote to info or widen.
Silent breakage — A metric goes to zero because its source died, and every rate threshold stays quiet because the sample gate isn't met. · Pair every rate line with a staleness or volume line on the same scope.
Moving the goalposts — Levels get loosened after each firing until nothing fires. · Every change is a version with a created_from reason; a re-fit needs labeled firings, not a bad mood.
Duplicate lines — Two thresholds watch the same metric on the same scope with different levels. · One threshold per metric and scope; the catalog rejects a second active one.
Mixed versions — A baseline built on an old prompt or detector version judges the new one. · Baselines reset when the watched thing's version changes; the line runs log only until it has history again.
Small-number panic — A rate line fires on 3 of 4 because the gate was set to zero. · Every row has a minimum sample. Rates under 30 units use Wilson intervals, not a fixed line.
data to capture
What each row and firing records.
record — Each threshold version (level, window, gate, action, owner, created_from) and each firing (value, sample, baseline, action taken).
label — Every firing labeled real, false alarm or unsure; every action labeled helped or didn't by +7 and +28 days.
trains — Re-fitted levels per scope, and a model that ranks firings by how likely they are to be real.
test — Run a proposed level silent beside the live one for 2 weeks and compare both against the labels.