Control limits and alert math

what it is, in industry terms

Statistical process control, for a growth pipeline.

Manufacturing has used control charts since the 1920s: measure a process, draw a center line and limits from its own history, and only intervene when a point lands outside the limits or a run of points drifts. The same idea separates a real drop in a play's reply rate from an unlucky week. Every rate in the pipeline is a share of units (replies per touch, real matches per label), so the workhorse here is the p-chart. Small counts get Wilson intervals; slow drifts get EWMA or CUSUM.

how it works

From history to a line.

1. History — 4 to 8 weeks same version
2. Clean it — drop incident days split by weekday?
3. Center line — p̄ or median
4. Limits — ±3σ act ±2σ watch
5. Gate — min sample n > 9(1−p̄)/p̄
6. Freeze — baseline row until re-fit

1. Take the history — Four to eight weeks of the metric on the current version of whatever it watches. Older history describes a different system.

2. Clean it — Drop days with known incidents (a dead source, a dispatcher outage). Note weekday patterns: a Monday dip in post reach isn't a drop.

3. Draw the center — The baseline rate over the clean history. For skewed costs, use the median.

4. Draw the limits — ±3σ for action, ±2σ for a watch note, with σ computed from this period's own n.

5. Set the gate — The smallest n where the lower limit clears zero, or 30 units, whichever is larger.

6. Freeze it — Store the baseline as a row. Re-fit on a schedule or when the watched version changes, never silently.

control limits for a rate

The p-chart: center line ± 3 standard errors.

center p̄ = baseline rate (e.g. replies ÷ touches over the last 4 to 8 weeks) sigma σ = √( p̄ · (1 − p̄) / n ) n = units in this period limits = p̄ ± 3σ (act) p̄ ± 2σ (watch)

Worked example: baseline rate p̄ = 8%, n = 30 units in the period, observed rate 3.0%.

8.0% — center line

22.9% — upper limit

noise — this period

With n = 30 the lower limit is below zero, so this line cannot catch a drop. It needs at least 104 units per period: widen the window or use a Wilson gate.

minimum sample

A lower limit below zero means the line is blind.

If p̄ − 3σ is under zero, no observed rate can ever fall below the limit: a drop to nothing still reads as noise. The lower limit only clears zero when n > 9 · (1 − p̄) / p̄. That's the real minimum sample for a floor on a rate, and it's why weekly limits on low rates rarely work for a small account.

small counts

Wilson intervals: honest ranges for a handful of units.

With 3 replies out of 20, "15%" hides how little is known. The Wilson interval gives the range the true rate plausibly sits in. A threshold on a small count fires only when the whole interval is past the line, not when the point estimate is.

Worked example: 3 successes out of 20 units, against a floor of 10%.

A fixed line at 10% on the point estimate alone would stay quiet. The interval says the data cannot tell yet.

false alarms add up

Many lines, checked often, will fire by chance.

A single 3-sigma line on a stable process fires by chance about 0.27% of the time per check (two-sided). Twenty lines checked daily is 140 checks a week. The worked example below shows how often at least one fires with nothing wrong.

Worked example: 20 active lines, each checked 7 times a week, at 3σ.

Assumes independent checks on stable processes. Real metrics are correlated, which lowers the count a little; it doesn't change the lesson.

the baseline object

Baselines are rows too.

A threshold compares against a frozen baseline, never a number recomputed at firing time. The alert stores the baseline it used.

a reply-rate baseline for one play

{
 "baseline_id": "bl_91a2…",
 "metric_id": "reply_rate_per_touch_7d",
 "scope": {
 "kind": "play",
 "id": "listen-and-reply"
 },
 "watched_version": "opener v5",
 "window": {
 "from": "2026-09-21",
 "to": "2026-10-18"
 },
 "excluded": [
 {
 "day": "2026-10-02",
 "why": "dispatcher outage"
 }
 ],
 "center": 0.11,
 "sigma_basis": "per-period n",
 "limits": {
 "act_sigma": 3,
 "watch_sigma": 2,
 "min_sample": 73
 },
 "method": "p-chart",
 "fitted_at": "2026-10-19T09:00:00Z",
 "supersedes": "bl_77f0…",
 "note": "example values"
}

methods

Which method for which metric.

illustrative example

Three lines, set properly on small numbers.

These are examples to show the arithmetic, not measured rates.

1. Reply rate on one play — Example: baseline 8%, about 30 touches a week. σ = √(0.08 × 0.92 / 30) ≈ 5 points, so the weekly lower limit is below zero: blind. A usable floor needs at least 104 touches, so read it on a rolling 4-week window (about 120 touches) or use EWMA weekly.

2. Detector precision — Example: baseline 71% over the last 50 labels. σ ≈ √(0.71 × 0.29 / 50) ≈ 6.4 points; the 3σ lower limit is about 52%. The catalog's starter floor of 60% sits inside that, so expect it to fire on noise sometimes; after a month of labeled firings, re-fit.

3. Stop requests — Example: a ceiling of 2 per 100 touches over 7 days, gated at 50 touches. This stays a fixed line on purpose: one bad week of complaints costs more than a false pause.

failure modes

How the math gets misused.

data to capture

What the math needs stored.

related pages

Where the math is used.