Improvement loop

what it is, in industry terms

Monitoring, alerting and a closed improvement loop.

Ops teams call the first half monitoring and alerting: a metric, a threshold, an alert. Quality engineers call the statistical version statistical process control: draw control limits from the metric's own history, and only act when a point falls outside them. Growth teams call the second half the improvement loop or build, measure, learn. This page joins the two halves, so that an alert can't just sit in a channel. Each firing either fixes something, opens a test, or gets labeled a false alarm, and the threshold learns from that label.

how it works

Eight steps, one loop.

The loop has no start. The daily snapshot measures, every threshold compares, a crossing triggers. Each step writes a row, so the loop can be replayed and audited.

1. Measure — The daily snapshot counts every rung, trigger, detector, play and campaign, plus the live event stream for guardrails.

2. Compare — Each active threshold reads its metric over its window. Below the minimum sample, it does nothing.

3. Trigger — A crossing writes an alert row and runs the threshold's action, unless the same threshold fired inside its cooldown.

4. Diagnose — The improvement agent pulls the slice behind the number: which detector, which play, which day, which version changed.

5. Hypothesis — One change, written into the hypothesis registry with the alert attached as evidence and an expected effect.

6. Test — Listen-side changes replay offline or run in shadow. Act-side changes get a randomized test with a holdout.

7. Decide — The readout applies the win rule set before launch: ship, revert or extend. The founder approves anything live.

8. Update — The baseline is recomputed on the new version, the threshold is re-fitted if needed, and the alert is labeled real or false.

the objects

A threshold, and the firing it produces.

Thresholds are data, not code. Each one is a versioned row, so changing a level is a logged decision with a reason, and old firings stay readable against the version that fired them.

a detector precision threshold

{
 "threshold_id": "thr.det.precision",
 "version": 3,
 "metric_id": "detector_precision_rolling",
 "stage": 5,
 "scope": {
 "kind": "detector",
 "id": null
 },
 "type": "floor",
 "window": "last 50 founder labels",
 "min_sample": 50,
 "level": {
 "value": 0.6,
 "unit": "share"
 },
 "severity": "warn",
 "action": "auto_shadow",
 "owner": "system",
 "cooldown": "7d",
 "created_from": null,
 "false_alarms": 0,
 "status": "active"
}

one firing of it

{
 "alert_id": "al_7f3c…",
 "threshold_id": "thr.det.precision",
 "version": 3,
 "scope_id": "det.public_pain.reddit",
 "value": 0.52,
 "sample": 50,
 "baseline": {
 "value": 0.71,
 "weeks": 4
 },
 "fired_at": "2026-10-20T13:05:00Z",
 "action_taken": {
 "kind": "auto_shadow",
 "hypothesis_id": "hyp_det_vocab_v4"
 },
 "acked_by": "system",
 "label": null,
 "outcome": null
}

action types

What a firing is allowed to do.

Ordered from least to most reach. The system can take the first four on its own because they stop or narrow something and are easy to undo. Anything that changes what a real person sees needs the founder.

cadences

Three speeds of checking.

tuning the thresholds themselves

A threshold is a hypothesis too.

Every starting level on this page is a rule of thumb, not a measured fact. The loop treats its own lines like any other knob: each one starts from a baseline, gets scored on its firings, and gets re-fitted on a schedule.

1. Start from a baseline — Before a threshold goes live, compute its metric over the last 4 to 8 weeks. Start the level at the rule of thumb or at the baseline's control limit, whichever is wider.

2. Run it silent first — New thresholds fire as log only for 2 weeks. Count how often they would have fired.

3. Label every firing — Each firing gets real, false alarm or unsure. The readout of any test it opened settles the label.

4. Re-fit monthly — Recompute the baseline and limits on the current version of whatever it watches. More than half false alarms widens the line; a labeled miss tightens it.

5. Version every change — A new level is a new version with a reason and a created_from learning id. Old firings stay tied to the version that fired.

6. Retire dead lines — A threshold that hasn't fired in 90 days and watches nothing critical becomes log only. Fewer, sharper lines beat a wall of alerts.

starter thresholds

Twenty lines to start with, across all twelve stages.

Levels are starting rules of thumb, not sourced numbers. Expect to re-fit most of them after the first month of firings. The full catalog, with every field, is on Threshold catalog.

illustrative example

One week of the loop, before launch.

Here is how the loop would handle that week. Every figure below is an illustration of the mechanics, not a measurement.

1. Monday: the pace line fires — thr.fun.pace: joins are behind the straight line to the goal. Severity warn, action Monday review. The founder opens the review with the rung chart beside it.

2. The weakest rung is brief offered, not found. One hypothesis: offer the brief in the first reply to Hot leads.

3. Tuesday: a detector floods — Illustration: a Reddit pain detector's matches jump past its upper control limit after a viral thread. The spike threshold alerts. Precision over the last 50 labels falls under 60%, so the precision threshold auto-shadows it and queues a vocabulary replay.

4. Wednesday: the replay reads out — The new vocabulary is replayed on stored matches with founder labels. Precision recovers without losing labeled-real matches, so the agent proposes the switch. The founder approves; the detector goes live on the new version.

5. Thursday: edit drift

   Illustration: the median edit distance on reply drafts rises by a third week over week. The agent replays the current prompt on the week's edited drafts and finds the opener is what gets rewritten. It proposes an opener test, which joins the queue behind the brief-offer test, since one live test per knob.

6. Sunday: label the week — Each firing is labeled. The flood was real; the edit drift turned out to be one long thread. One false alarm goes against that threshold, and nothing is re-fitted yet.

data to capture

Firings are labels.

go deeper

The four parts of the loop, and its neighbours.