Trigger to action runbooks

what it is, in industry terms

Incident runbooks, extended into experiments.

Reliability teams write runbooks so whoever is paged knows the first five minutes without thinking. Growth teams rarely do, so metric drops get argued about instead of handled. These runbooks borrow the incident shape (contain, diagnose, fix, review) and add one step: every diagnosis that isn't plain breakage ends in a hypothesis, so the fix is tested, not guessed.

how a runbook runs

Contain first, then understand, then change one thing.

1. Fire — threshold crosses
2. Contain — pause · shadow alert (system)
3. Diagnose — slice the data (agent)
4. Fix or — hypothesis breakage → fix drop → test
5. Approve — founder, for anything live
6. Close — label firing log outcome

the runbook object

Runbooks are versioned like everything else.

the detector quality runbook

{
 "runbook_id": "rb.detector_quality",
 "version": 2,
 "fires_on": [
 "thr.det.precision",
 "thr.det.spike"
 ],
 "severity": "warn",
 "steps": [
 {
 "who": "system",
 "do": "move detector to shadow",
 "within": "1 tick"
 },
 {
 "who": "agent",
 "do": "sample 20 recent matches; compare vocab hits vs labels",
 "within": "24h"
 },
 {
 "who": "agent",
 "do": "queue offline replay of vocab v-next on stored matches",
 "within": "24h"
 },
 {
 "who": "founder",
 "do": "approve go-live if precision recovers and recall holds",
 "within": "72h"
 }
 ],
 "test_template": "hyp_template.detector_vocab",
 "close_when": "precision ≥ floor on 50 new labels, or detector retired",
 "logs": [
 "alert_id",
 "replay_id",
 "decision",
 "label"
 ]
}

runbooks

Ten trigger families, ten runbooks.

Detector quality — warn · fires on: Precision under its floor, or a match spike past the control limit

1. systemMove the detector to shadow. It keeps logging, stops filling the list.
2. agentPull 20 recent matches; mark which vocabulary terms or rule fired on the ones labeled not real.
3. agentDraft vocab or rule v-next and queue an offline replay on stored matches with labels.
4. founderApprove go-live if precision clears the floor and labeled-real matches aren't lost.

Source stale or failing — warn · fires on: Newest item older than max staleness, or over 10% of runs partial or error

1. systemMark the source stalled or degraded; alert. Detectors on it are marked as missing input, so their silence isn't read as no demand.
2. agentCheck the last good run, the error text and whether the feed URL or API changed.
3. founderDecide: fix, swap to another source, or retire. A swap keeps the same row shape.

Reputation guardrail — critical · fires on: Stop requests over 2 per 100 touches, a touched holdout, or a spam report

1. systemPause the campaign immediately.
2. agentList the touches that drew stops: play, step, template, channel, time.
3. founderRead them. Decide: kill the step, rewrite the template, or narrow the segment.
4. founderApprove a restart only with the change in place.

Play underperforming — warn · fires on: Reply rate per touch below the play's lower control limit (4-week window)

1. agentSplit the drop by step, channel, lead tier and trigger age. Rule out a source or lead-mix change first.
2. agentIf the mix is stable, draft an opener test: current opener vs one challenger, same play.
3. founderApprove the test; it runs with a holdout share and a pre-set win rule.

Draft drift — warn · fires on: Median edit distance up a third week over week, or unedited share down 25%

1. agentGroup the week's edits by edit locus (hook, body, ending) to see where the founder rewrites.
2. agentQueue an offline replay: current prompt vs a v-next tuned on the edits, scored against what was actually posted.
3. founderApprove v-next if the replay brings drafts closer to what was posted.

Lead ranking off — warn · fires on: Top-10 reply rate below its lower control limit 2 weeks running, or founder splits over 5% of merges

1. agentReplay the scorer on frozen features: does a reweighted version rank past repliers higher?
2. agentFor merges: list the split pairs and the rule that joined them; tighten and replay.
3. founderApprove the new scorer or merge rule.

Funnel rung below limits — warn · fires on: One rung's conversion below its lower control limit 2 weeks running

1. agentConfirm the rung's input volume and mix didn't change; find the step that feeds it.
2. founderMonday review: make it the constraint for the week.
3. agentDraft one hypothesis aimed at that rung.

Behind the goal pace — warn · fires on: Waitlist joins behind the straight line to the goal

1. systemPut the pace line and the rung chart at the top of the Monday review.
2. founderPick the weakest rung as the week's constraint.
3. agentOpen one hypothesis on that constraint; park other open ideas.

Readout or label overdue — info · fires on: A test past its judge-by date, a campaign past its readout date, or a firing unlabeled after 7 days

1. systemNudge the owner with the one action needed.
2. agentPre-compute the readout so it only needs a decision.

Threshold crying wolf — info · fires on: Over half of a threshold's last 10 firings labeled false alarm

1. agentRe-fit the level from the last 8 clean weeks; draft it as a new version.
2. agentRun the new level silent beside the old for 2 weeks.
3. founderApprove the swap if it would have caught the real firings with fewer false ones.

illustrative example

Nov 13: Reddit RSS goes quiet.

Say the Reddit feeds you read stop on Nov 13. Here is the source-stale runbook running on that day, written ahead of time.

1. Fire — thr.src.stale: the Reddit sources' newest item passes max staleness. The run-error line may fire first, as fetches start failing.

2. Contain — The system marks the sources stalled. Reddit detectors are flagged missing input, so the drop in Reddit signals isn't read as Reddit demand falling.

3. Diagnose — The agent confirms the feeds return nothing, not an error page, and that the end date matches the known Nov 13 cutoff.

4. Decide — The founder picks the replacement source. It runs in shadow first, writing rows in the same shape so detector history stays comparable.

5. Close — Once the replacement's matches look like the old ones on labels, it goes live. The firing is labeled real; the runbook logs downtime and leads missed.

failure modes

How runbooks fail.

data to capture

Every runbook run is a training example.

related pages

Where runbooks hand off.