listen · detectors
Testing a detector
Shadow runs, hand tests and yield.
In Detectors
Testing a detector: hand tests, shadow runs, precision samples and yield per dollar.
A detector that hasn't been tested on real data is a guess. This is the protocol every monitoring rule goes through, from a 30-day hand search to a shadow run beside the live version, then weekly samples and a monthly yield readout that decides whether it keeps running.
Triggers → Detector catalog → Testing → Signal catalog
Write the gate first — The pass rule is set before anyone reads the labels.
Humans label, the model is tested — Never grade the judge with the judge.
Yield beats precision — A precise detector that never leads to a reply gets retired.
what it is
Offline evaluation, shadow deployment, then monitoring.
Industry names: offline evaluation on labeled samples, shadow deployment (or dark launch) beside the live version, precision at k sampling, and monitoring for drift. The same protocol tests a new detector, a new rule version, a new judge prompt or a new source.
before build — Is it real? · Hand search, count, a precision read and a miss log. About an hour each.
before live — Is it better? · A shadow run on the same items as live, with the gate written in advance.
while live — Is it still worth it? · Weekly samples, a silence alarm and a monthly yield readout per dollar.
the lifecycle
Every detector moves through gates.
- Draft — rule v1
- Hand test — 30 days, by hand
- Precision read — 20+ labels
- Shadow — beside live, silent
- Live — emits signals
- Readout — monthly yield
- Retire — or tune gate: real matches gate: half pass gate: no worse gate: yield per $ weekly sample: false matches become exclusions, version + 1, back through shadow
the test record
Every test is a record.
Kept with the detector so its history explains its current version. The sample is the Reddit-successor shadow test, with blanks where numbers will go.
| Field | Type | What it holds |
|---|---|---|
test_id | string | One evaluation of one detector version. |
detector_id + version | ids | What was tested. |
stage | enum | hand_test | precision_read | shadow | weekly_sample | readout | retire_review |
window | dates | The items' date range. |
items | integer | Items read or seen. |
labels | list | Per item: real, wrong or unsure, by whom, with the reason. |
metrics | object | Precision, margin, yield, cost per real signal, agreement. |
gate | rule | The pass rule, written before the test ran. |
decision | enum | pass | tune | retire, with a note. |
cost | object | API, model and founder minutes spent on the test. |
one test, as data
{
"test_id": "t_2026_10_20_reddit_successor",
"detector_id": "det.tried-it-and-it-failed.hn",
"version": 1,
"stage": "shadow",
"window": [
"2026-10-20",
"2026-11-03"
],
"items": "(fill in)",
"labels": [
{
"item": "hn:45123456",
"label": "real",
"by": "founder",
"reason": "Show HN, 2 points, maker says no signups"
}
],
"metrics": {
"precision": "(fill in)",
"margin_95": "(fill in)",
"yield_per_month": "(fill in)",
"cost_per_real_signal_usd": "(fill in)",
"agreement_with_live": "n/a (new source)"
},
"gate": "precision >= 0.5 on >= 50 labels AND yield >= 20/month (set Oct 20)",
"decision": null
}metrics
What gets measured, and how.
| Metric | Formula | Read when |
|---|---|---|
| Precision | real matches ÷ matches labeled, on a random sample | before shadow, weekly |
| Recall (estimate) | caught ÷ (caught + misses found another way) | monthly, from the miss log |
| Yield | real signals per month | before build, monthly |
| Cost per real signal | (API + model + founder minutes × hourly value) ÷ real signals | before live, monthly |
| Signal to lead | leads ÷ real signals | monthly |
| Reply rate | replies within 7 days ÷ touches, per detector | monthly, once touch ids exist |
| Join or pay rate | waitlist joins or payments ÷ touches, per detector | monthly; small numbers, read with care |
| Shadow agreement | same verdict ÷ items seen by both versions | during shadow |
how many labels
Small samples have wide margins.
The margin at 50% precision is the widest case (1.96 × √(0.25 ÷ n)). Use it to decide how much a precision number can tell you.
| Labels read | Margin at 50% precision (95%) | Good for |
|---|---|---|
| 20 | ± 22 points | a first look: is it mostly right or mostly wrong? |
| 50 | ± 14 points | go or no-go for shadow |
| 100 | ± 10 points | comparing two rule versions with a big difference |
| 200 | ± 7 points | a judge eval set; small differences between versions |
| 400 | ± 5 points | fine-tuning decisions |
the checks
Fourteen checks, from hand search to retirement.
before build
Hand search
- how
- Run the match by hand on the source for the last 30 days.
- pass if
- Any real matches at all.
- catches
- A trigger that doesn't show up where you thought.
before build
Count
- how
- Count real matches per month.
- pass if
- 20 or more makes it a channel (rule of thumb).
- catches
- Building automation for 3 people a month.
before build
Precision read
- how
- Read 20 matches; mark right person, in the moment, fits the ICP.
- pass if
- At least half pass (rule of thumb).
- catches
- Noisy phrases; vendors; wrong ICP.
before build
Miss log
- how
- Write down real people you found another way that the rule missed.
- pass if
- Each miss becomes a phrase or a fix.
- catches
- Vocabulary written by the founder, not the buyer.
before build
Exclusion check
- how
- Confirm not-if lines and lead rules throw out the right people.
- pass if
- Zero excluded people in the sample.
- catches
- Pitching existing customers or competitors.
before build
Freshness check
- how
- Check the item's own date is read, not the capture date.
- pass if
- Every match has posted_at or event_date.
- catches
- Old threads treated as live.
before build
Dedupe check
- how
- The same post or job seen twice collapses to one signal.
- pass if
- No duplicate people in a day's output.
- catches
- Double outreach.
before live
Shadow run
- how
- New version runs beside live on the same items, emitting nothing.
- pass if
- Agreement and precision at least as good as live, on the same weeks.
- catches
- Regressions from a prompt or vocabulary edit.
before live
Judge agreement
- how
- Compare judge verdicts to founder labels on the labeled set.
- pass if
- Agreement high enough that you'd accept its errors (set before you look).
- catches
- A judge that sounds confident and is wrong.
before live
Cost check
- how
- Measure API and model cost per real signal.
- pass if
- Under the value of a reply (set per product).
- catches
- A detector that burns credits on noise.
Weekly sample
- how
- Read a sample of what it caught; new false matches become exclusions.
- pass if
- Precision stays within the margin of the pre-live read.
- catches
- Drift as the source or audience changes.
Silence alarm
- how
- Alert when a detector that normally fires goes quiet.
- pass if
- Fires within its usual range.
- catches
- A source changed format or access ended.
monthly
Yield readout
- how
- Signals to leads to replies to joins to payments, per detector, per dollar.
- pass if
- Ranks detectors; the top ones get more budget.
- catches
- Detectors that are precise but never convert.
monthly
Retire review
- how
- Switch off detectors whose signals never become a reply after a fair run (count set in advance).
- pass if
- Retire or tune, with the record kept.
- catches
- A library full of dead rules.
illustrative example
Replacing the Reddit feed before Nov 13.
week 1 — Hand test Show HN · Search the last 30 days of Show HN for low-point launches with a maker who says it flopped. Count real ones.
week 1 — Write the gate first · Pick the pass rule before reading labels, e.g. at least half pass on 50 labels and 20+ a month. Then read.
weeks 2 to 3 — Shadow it · Run the new detector in shadow next to the live Reddit lane; same row shape, emits nothing.
by Nov 13 — Promote or keep looking · If it passes, make it live before the Reddit feed ends, so detector history continues in the same shape.
no numbers yet
The sample record above has blanks on purpose. Fill them from the hand test; don't estimate.
failure modes
How testing goes wrong.
| Failure | What it looks like | How to catch it |
|---|---|---|
| Grading its own homework | The judge's verdicts used as the labels for the judge. | Labels come from the founder (or a second reader), never from the model under test. |
| Too few labels | "9 of 12 were right" treated as 75% precision. | Report the margin; 20 labels is about ±22 points. |
| Only reading matches | Precision looks great; recall is unknown. | Keep a miss log of people found another way. |
| Moving the goalposts | The pass rule is chosen after seeing the result. | Write the gate before the test; store it with the test record. |
| Shadow on different weeks | Comparing a new version this week to the old one last month. | Shadow runs on the same items at the same time. |
| Silent source death | A detector returns zero and nobody notices. | Silence alarm on every live detector. |
| Cost creep | Judge calls grow with a broader rule. | Cost per real signal in every readout. |
data to capture
Keep every test, and every label.
The labeled sets from testing become the eval sets every later version is replayed against.
record
- the test record: detector version, stage, window, gate, decision
- every label with who gave it and why
- shadow verdicts as rows, next to live ones
- cost per run: API, model, founder minutes
label
- real, wrong or unsure per matched item
- misses: people found another way
- later outcomes: lead, reply, join, payment
trains
- the match judge and its regression set
- precision and yield priors for new detectors on the same source
- a retire model: which early numbers predict a dead detector
test
- new version vs live in shadow on the same weeks
- judge prompt v2 replayed on the stored labeled set
- 20-label vs 50-label gates: how often the early call was wrong