all guides

Learn what worked

Experiments, metrics and the Monday read. Most small teams run a test, get a number and ship whichever side looked ahead that day. This is the method for knowing what actually worked: size a test before you start, read it once, define every number once, and end each week with one change. Everything runs in your browser, and nothing you type leaves it.

free library

make it yours

showing an example product, shipnote. paste your site and we fill it in, or type yours once, and every template, search and message across the libraries switches to it.

the free autofills on this connection are used up for this hour

a shared connection counts as one, so this can happen on your first try. it resets within the hour, or join the waitlist and get 5 more today.

one email when vibedraft opens, and a short build note now and then. one click to leave.

Get the vibedraft newsletter, launch updates, and a chance to become a beta tester. Unsubscribe any time.

Measure, compare, trigger, test, decide.

Analysis stops being something you remember to do on Sundays and becomes a loop that tells you where to look. Walk round it: who does each step, and what it writes down.

the loop
1measure2compare3trigger4diagnose5hypothesis6test7decide8updateevery row versionedevery firing labelled
Step of the loop

1. Measure

done by the system

A daily snapshot counts every rung, trigger, detector, play and campaign, plus the live event stream for guardrails.

Next: .

Not a dashboard. Dashboards wait to be read; thresholds come to you. Anything that changes what a real person sees needs you to approve it.

Seventeen tests worth running first.

Each card is a complete test minus the dates. Open one to pre-register it: the hypothesis sentence, the units each arm needs at your volume, the six-week warning, and a markdown copy to keep with the test. Three rules hold for all of them: one live test per knob, size it before launch, read it out win or lose.

test library
Phase
Mode

17 of 17 tests

exp.next moves.opener-proofRandomized A/B

Proof opener vs plain headline

stage 10 · next moves · act

if
A number from your own work in the first line earns more reach and replies than a plain claim.
unit
post
arms
control: plain claim, variant: proof opener, same body
metric
impressions at 24h; replies
guard
negative feedback
size
Pairs well with a threshold rule: turn each post's reach into yes or no against a line you set before launch. Two pairs of posts is a direction, not a finding.
exp.plays.reply-firstRandomized A/B

Reply first vs DM first

stage 8 · plays · act

if
Answering in public before a DM earns more replies than a cold DM on a fresh signal.
unit
lead
arms
control: DM first, variant: public reply, DM only after they answer
metric
replied within 7 days
guard
stop requests, blocks
size
About 144 leads per arm to see 5% to 15%.
exp.next moves.send-timeRandomized A/B

Best hour vs off hour

stage 10 · next moves · act

if
Posts in the account's best measured hour beat the same kind of post three hours later.
unit
post
arms
control: off hour, variant: best hour from your engagement matrix
metric
engagement score at 24h
guard
none
size
Randomize the slot per draft and keep topics balanced across arms.
exp.signals.judge-v2Offline replay

Judge prompt v2

stage 6 · signals · listen

if
A judge prompt with three worked examples agrees with your labels more often.
unit
stored match
arms
v1 verdicts against v2 verdicts on the same labelled matches
metric
precision on the top 20 a day
guard
recall must not drop
size
Needs about 200 labelled matches. Free to run, no live risk.
exp.detectors.vocab-v2Shadow

Detector vocabulary v2

stage 5 · detectors · listen

if
Adding buyer phrases from replies raises real matches without raising noise.
unit
raw item
arms
live v1, shadow v2 on the same feed
metric
matches labelled real per 100
guard
run cost per real match
size
Two weeks of shadow, then a hand check of 50 disagreements.
exp.moments.fresh-windowOffline replay

Freshness window 24h vs 72h

stage 4 · moments · listen

if
Signals older than 24 hours reply far less, so the window should be shorter.
unit
signal
arms
bucketed by age at first touch: 0 to 24h, 24 to 72h
metric
reply rate by age bucket
guard
none
size
Read the decay curve offline first, then randomize which cohort gets touched late.
exp.campaign.offer-briefRandomized A/B

Offer: free thing first vs ask for the signup

stage 9 · campaign · act

if
Offering something useful first, a free teardown or a short brief, gets more signups than asking for the signup directly.
unit
lead
arms
control: signup link, variant: the free thing first, signup after
metric
signed up within 14 days
guard
stop requests; your minutes
size
Signups are rare. Test only with a big expected gap, or use replies as a proxy metric.
exp.campaign.holdout-leadHoldout lift

Holdout lift on a lead-to-signup campaign

stage 9 · campaign · act

if
The campaign causes signups that would not happen anyway.
unit
lead
arms
90% treatment, 10% holdout gets no touch
metric
signup rate, treatment minus holdout
guard
none
size
Needs the most units of any test, and the holdout is the small arm so it sets the size. Run it continuously and read it monthly.
exp.plays.comment-kwRandomized A/B

Comment a keyword vs link in reply

stage 8 · plays · act

if
"Comment X and I'll send it" earns more replies and joins than a link under the post.
unit
post
arms
control: link in reply, variant: comment-a-keyword
metric
replies per post; joins per post
guard
replies you cannot answer within a day
size
Alternate in randomized order. Six to ten posts per arm finds only a large gap.
exp.next moves.article-longRandomized A/B

Long article vs long post

stage 10 · next moves · act

if
The article format reaches further than the same text as a long post.
unit
piece of content
arms
control: long post, variant: article, same text
metric
impressions at 24h; profile visits
guard
none
size
Same text in both arms isolates the format, not the topic.
exp.next moves.amplifyRandomized A/B

Plain repost vs quote

stage 10 · next moves · act

if
A quote from your personal account lifts the brand account's post more than a plain repost.
unit
brand-account post
arms
nothing, a plain repost, a quote
metric
engagements at 24h on the brand post
guard
none
size
Block-randomized: each arm once per 3-day block, so a slow week hits every arm equally. Three arms need a stricter rule than two.
exp.opportunities.scorer-v2Offline replay

Lead scorer v2

stage 7 · opportunities · listen

if
Weighting signal freshness higher puts more repliers in the daily top 10.
unit
lead
arms
v1 rank against v2 rank over past leads with known replies
metric
replies among the top 10 a day
guard
no tier inflation
size
Offline first on frozen features, then a randomized half of the daily list.
exp.plays.recommenderOff-policy estimate

Playbook recommender v1

stage 8 · plays · act

if
A learned play choice beats the hand-written default on reply rate.
unit
move
arms
current policy against new policy, estimated on logged moves
metric
estimated reply rate
guard
contact-policy violations = 0
size
Impossible until the odds of each move are logged. Then estimate before any live slice.
exp.icp.icp-cutOffline replay

Cut a weak fit line

stage 3 · icp · understand

if
Dropping a fit line that never predicted payment keeps accuracy and grows the list.
unit
past lead
arms
with the line, without it
metric
agreement with your accept and with paid
guard
precision must not drop
size
Runs on the reviewed leads you already have.
exp.brief.brief-v2Offline replay

Brief extractor v2

stage 2 · brief · understand

if
A second pass that quotes the source for every field reduces your corrections.
unit
stored site
arms
v1 brief against v2 brief on the same sites
metric
fields kept unedited
guard
cost per brief
size
Every corrected field is a test case. About 50 is enough to start.
exp.funnel.hero-copyRandomized A/B

Waitlist page hero copy

stage 11 · funnel · learn

if
Naming the pain in the headline beats naming the product.
unit
visitor
arms
control: product headline, variant: pain headline
metric
waitlist join rate
guard
bounce rate
size
Visitor tests need traffic: about 1,164 visitors per arm to see 2% to 4%.
exp.next moves.steps-2v4Randomized A/B

Two steps vs four steps

stage 10 · next moves · act

if
A shorter sequence loses few replies and cuts stop requests.
unit
lead
arms
control: 4 steps, variant: 2 steps
metric
replied within 14 days
guard
stop requests
size
Read the guardrail as closely as the metric.

Is it real?

Four checks you can run on your own numbers. A significance check with the verdict table, a peeking simulator that shows why daily looks fool you, the false-win odds of a pre-set rule, and a control chart that tells a real drop from noise.

the lab

Did the variant win?

your two arms (example numbers, replace with yours)

Replied, joined, cleared the line.
People, posts or visitors in the arm.
Set before launch, not after.
From your pre-registration. Short of it, the test is underpowered.
Bayesian prior, from your past tests (optional)
Blank means flat. Use a past learning on the same channel and account size.
In pseudo-units. 2 is flat; 20 to 50 is a firm past result.

is it real?

Verdict: ship.

The interval sits above zero and the plain win rule passes.

Roll out the arm, bump the version, and write the learning with its effect and interval.

8% vs 16%control vs variant
+0.6 to +15.595% range, points, variant minus control
0.033p, two-proportion z-test
1.000split check (under 0.001 voids it)
98%chance the variant is better
+1.8 to +14.290% range, points
0.0expected loss if you ship it, points

150 and 150 units. Small samples give wide answers, honestly. Inconclusive stays inconclusive: do not promote a near miss to a finding.

the decision table, read in this order

  1. shipran cleanly: yes. guardrails: none broken. primary: interval above zero and the win rule passes.Roll out the arm, bump the version, and write the learning with its effect and interval.
  2. revertran cleanly: yes. guardrails: none broken. primary: interval below zero.Keep the control. Log the loss; it is as useful as a win.
  3. extendran cleanly: yes. guardrails: none broken. primary: interval spans zero and still includes an effect worth acting on.Extend once, to a size set in the record before launch. If still unclear, close it as inconclusive.
  4. killran cleanly: yes. guardrails: none broken. primary: interval spans zero and excludes any effect worth acting on.The knob does not matter at this size. Keep whichever arm is cheaper and move to the next constraint.
  5. revertran cleanly: any. guardrails: a guardrail broke. primary: not read.Stop now, even mid-test. Guardrails are the one allowed early look.
  6. voidran cleanly: no: split mismatch or broken exposure. guardrails: any. primary: not read.Fix the assignment or logging bug, then rerun from scratch. Never repair a void test with maths.
  7. underpoweredran cleanly: no: sample not reached by judge-by. guardrails: none broken. primary: not read.Extend once if the plan allows; otherwise close it as underpowered. Not a loss, not a win.

Why peeking breaks a test

peeking simulator: 2,000 A/A tests, both arms identical

...look once, at the end
...stop at the first of 10 looks under p 0.05
once
10 looks

Running in small slices so the page stays responsive. The black line is the promised 5%. Look every day and stop at the first "win", and you will win by luck far more than 5% of the time. Small samples and low rates make the test a little conservative, so "once" can land under 5%. Seeded: everyone sees the same numbers for the same settings.

How often does your rule win by luck?

a win rule is a test too: know its false-win rate

12.5%chance one version wins when both are the same (2 posts each)
0.78%if the same version must win on two independent pairs
2 to 5
1 to 6
78%chance some arm takes 2 of 3 blocks when all 3 arms are identical
Threshold rule, 2 posts per versionused asBoth posts clear the line and the other version's don'twins by luckat most 12.5% (when each post clears half the time)stronger3 or 4 posts per version, or the same winner in a second pair
Block majority, 3 arms, 2 of 3 blocksused asAn arm beats both others in at least 2 of 3 blockswins by luckabout 78% (21 of 27 block outcomes produce a winner)stronger3 of 3 blocks (about 11%), or more blocks with a margin over the runner-up
Rank the top lanesused asTop 4 of 6 topic lanes by pooled scorewins by lucknot pass or fail: any lane is in the top 4 by chance 2 times in 3strongertreat it as a ranking to follow up, then confirm the winner with a 2-arm test
Two-proportion z-test, one lookused asThe default for yes or no metricswins by luck5%, by designstrongerfine as is; add a guardrail-only interim
The same test, 10 looks, stop at the first winused asWhat happens without a rulewins by luckroughly 15 to 20% (run the simulator)strongera planned sequential design, or one look

Read the amplification-style rule, an arm beating both others in 2 of 3 blocks, as a direction and not a finding: it picks a winner about 78% of the time even when all three arms are the same. A rule with a false-win rate above 10% caps your confidence at suggestive.

ways to look safely

  • Do not look at the primary metric. Watch guardrails only: stop requests, spam reports, credits. Stopping for harm is always allowed.
  • Plan the looks. A group-sequential design spends the 5% across, say, 3 planned looks, with a very strict line early and close to the normal line at the end.
  • Use an always-valid test. It gives p-values you can check any time, at the cost of needing a bit more data.
  • Read a Bayesian posterior without a stop rule, on the judge-by date only. Bayesian numbers do not make peeking free if you still stop on the first good-looking day.

Is this drop real or noise?

is this drop real or noise? a p-chart

Touches, posts, visitors.
8%centre line
0.6%lower limit (3 sigma)
15.4%upper limit (3 sigma)
watchthis period
UCLpLCLthis period 2%

Usable: a drop below 0.6% on 120 units is a real signal, not noise. Act at 3 sigma; at 2 sigma, watch.

the blind rule: units needed before the lower limit clears zero

2% usual rateat least 442 units12 weeks
5% usual rateat least 172 units5 weeks
8% usual rateat least 104 units3 weeks
10% usual rateat least 82 units3 weeks
15% usual rateat least 52 units2 weeks
20% usual rateat least 37 units1 week or less
30% usual rateat least 22 units1 week or less
50% usual rateat least 10 units1 week or less

The lower limit only clears zero when n is bigger than 9 times (1 minus the rate) over the rate.

small counts: 3 replies out of 20 hides how little is known

Not enough evidence yet. Keep collecting.

15% observed, 95% range 5.2% to 36%. A fixed line on the point estimate alone would stay quiet; the range says the data cannot tell yet.

Every number, defined once.

Twenty-six metrics, each with the question it answers, its formula, its window and the smallest count worth a rate. If you cannot write the question, drop the metric.

metric dictionary
One definition. One id, one formula, one window. A dashboard, the Monday read and a test readout can then never disagree.
Write the question first. If you cannot write the question the metric answers, drop the metric.
Name the window. A reply rate without a window changes every day as late replies arrive. Count only closed windows.
Counts below the minimum. Rates on six people are noise. Under the minimum sample, show the count and no rate.
Version it. A changed formula, window or event is a new version, so old readouts stay comparable.
Kind

26 of 26 metrics

stage 4 · trigger 1

met.detection_lagdiagnosticlower is better

Detection lag

How fresh is what we catch?

formula
median hours from the source post's own timestamp to capture
unit
hours, per signal
window
at capture, read daily
reads
item_matched
min sample
30 units, below that show the count

stage 5 · detector 2

met.signal_precisionprimaryhigher is better

Signal precision

Of what a detector catches, how much is real?

formula
matches the founder labels real ÷ matches labeled
unit
a rate, per detector
window
30 days, read weekly
reads
match_labeled
min sample
30 units, below that show the count
met.yield_per_dollarprimaryhigher is better

Yield per detector dollar

Which detectors earn their cost?

formula
leads (or replies) from a detector's signals ÷ its run cost in dollars
unit
a rate, per detector
window
30 days, read weekly
reads
detector_run_completed, lead events
min sample
30 units, below that show the count

stage 6 · signal 1

met.judge_agreementdiagnostichigher is better

Judge agreement

Does the match judge agree with the founder?

formula
matches where judge verdict = founder label ÷ labeled matches
unit
a rate, per judge version
window
30 days, read weekly
reads
match_judged, match_labeled
min sample
30 units, below that show the count

stage 7 · lead 1

met.lead_accept_ratediagnostichigher is better

Lead accept rate

Does the founder agree the lead fits?

formula
leads accepted ÷ leads reviewed (accept, skip, never)
unit
a rate, per lead
window
at review, read weekly
reads
lead_reviewed
min sample
30 units, below that show the count

stage 9 · campaign 2

met.offer_to_joinprimaryhigher is better

Waitlist offer to join

When we point someone at the waitlist, do they join?

formula
waitlist sign-ups ÷ leads offered the waitlist
unit
a rate, per lead
window
14 days, read weekly
reads
waitlist offered, signup_created
min sample
30 units, below that show the count
met.campaign_liftprimaryhigher is better

Campaign lift

What did the campaign cause?

formula
conversion in the treatment arm − conversion in the holdout
unit
a rate, per campaign
window
campaign window, read weekly
reads
lead_assigned, signup_created
min sample
30 units, below that show the count

stage 10 · sequence 8

met.reply_rate_7dprimaryhigher is better

Reply rate per touch

Do the messages get answered?

formula
touches with a reply within 7 days ÷ touches sent
unit
a rate, per touch
window
7 days, read weekly
reads
touch_sent, reply_received
min sample
30 units, below that show the count
met.positive_reply_rateprimaryhigher is better

Positive reply rate

Are the answers good ones?

formula
replies classed positive or question ÷ touches sent
unit
a rate, per touch
window
7 days, read weekly
reads
reply_received (class)
min sample
30 units, below that show the count
met.time_to_first_replydiagnosticlower is better

Time to first reply

How fast do people answer?

formula
median hours from touch sent to its first reply
unit
hours, per touch
window
7 days, read weekly
reads
touch_sent, reply_received
min sample
30 units, below that show the count
met.stop_rateguardraillower is better

Stop requests per 100 touches

Are we annoying people?

formula
stop, block or report events ÷ touches sent × 100
unit
a rate, per touch
window
7 days, read daily
reads
negative_feedback
min sample
30 units, below that show the count
met.drafts_uneditedprimaryhigher is better

Drafts shipped unedited

How often is the draft good enough as written?

formula
drafts decided shipped ÷ drafts decided (shipped, shipped edited, discarded)
unit
a rate, per draft
window
at decision, read weekly
reads
draft.decided
min sample
30 units, below that show the count
met.edit_distance_mediandiagnosticlower is better

Median edit distance

How much does the founder rewrite?

formula
median normalized edit distance of drafts shipped edited, split by edit locus (hook, body, ending)
unit
a rate, per draft
window
at decision, read weekly
reads
draft.decided
min sample
30 units, below that show the count
met.time_to_decidediagnosticlower is better

Time to decide

How long does a draft wait on the founder?

formula
median seconds from draft generated to decided
unit
seconds, per draft
window
at decision, read weekly
reads
draft.generated, draft.decided
min sample
30 units, below that show the count

stage 11 · funnel 10

met.waitlist_joins_weeklyprimaryhigher is better

Waitlist joins per week

Are we on pace for the goal?

formula
count of waitlist sign-ups per ISO week, against a straight line to your goal
unit
a count, per week
window
7 days, read weekly
reads
sign-up created (waitlist)
min sample
none, a plain count
met.rung_conversiondiagnostichigher is better

Rung conversion

What share of people move from one rung to the next?

formula
people reaching rung n+1 ÷ people reaching rung n, among those entering rung n in the window
unit
a rate, per rung, week
window
14 days, read weekly
reads
ladder rung events
min sample
30 units, below that show the count
met.impressions_24hdiagnostichigher is better

Impressions at 24 hours

How far did a post travel?

formula
impressions from the 1,440-minute snapshot
unit
a count, per post
window
24 hours, read daily
reads
post.published, post_snapshots
min sample
none, a plain count
met.profile_clicks_per_100primaryhigher is better

Profile clicks per 100 impressions

Did the post make people want to know who wrote it?

formula
profile clicks ÷ impressions × 100, at the 1,440-minute snapshot
unit
a rate, per post
window
24 hours, read weekly
reads
post_snapshots
min sample
30 units, below that show the count
met.early_forecast_errordiagnosticlower is better

30-minute forecast error

Does the 30-minute number predict the 7-day number?

formula
error of a 7-day impressions forecast made from the 30-minute snapshot
unit
a rate, per post
window
7 days, read weekly
reads
post_snapshots
min sample
30 units, below that show the count
met.visit_to_signupprimaryhigher is better

Visit to sign-up

Does the page convert the people we send?

formula
sign-ups ÷ visits that carry a touch id
unit
a rate, per visit
window
7 days, read weekly
reads
visit_started, signup_created
min sample
30 units, below that show the count
met.activation_7dprimaryhigher is better

Activation within 7 days

Do new sign-ups do the thing that predicts paying?

formula
sign-ups with the activation action within 7 days ÷ sign-ups
unit
a rate, per account
window
7 days, read weekly
reads
signup_created, activation_reached
min sample
30 units, below that show the count
met.trial_to_paidprimaryhigher is better

Trial to paid

Do trials become payments?

formula
accounts with invoice.paid within 14 days of trial start ÷ trials started
unit
a rate, per account
window
14 days, read weekly
reads
invoice.paid
min sample
30 units, below that show the count
met.churn_monthlyguardraillower is better

Monthly churn

Are customers leaving?

formula
subscriptions canceled in the month ÷ active at the start of the month
unit
a rate, per month
window
1 month, read weekly
reads
customer.subscription.deleted
min sample
30 units, below that show the count
met.payment_recoverydiagnostichigher is better

Failed payment recovery

Do failed payments get recovered?

formula
failed invoices later paid ÷ failed invoices
unit
a rate, per invoice
window
14 days, read weekly
reads
invoice.payment_failed, invoice.paid
min sample
30 units, below that show the count

stage 12 · loop 1

met.founder_minutes_per_joinguardraillower is better

Founder minutes per join

What does each join cost in your time?

formula
founder minutes spent (reviewing, editing, sending) ÷ joins
unit
minutes, per week
window
7 days, read weekly
reads
draft.decided, lead activity
min sample
30 units, below that show the count

Fifteen minutes, one change.

Ten sections, each with the numbers to read and what to do about them, as a form that comes out as markdown. It ends with exactly one change; everything else waits a week. Below it, the clocks the numbers run on.

the Monday read
  1. 1Freeze the weekTake a snapshot at the end of Sunday. The read uses only the snapshot, so reading it twice gives the same numbers.
  2. 2Apply minimum samplesBelow a metric's minimum sample, write the count and no rate.
  3. 3Compare to its own pastThis week against the 4 weeks before, per metric. Never against a benchmark from someone else's product.
  4. 4Mark what is dueAny experiment with a judge-by date this week gets its verdict now.
  5. 5Decide one thingWrite the constraint, the hypothesis and the change. Put the judge-by date and the win rule in writing before it ships.
  6. 6Schedule the readoutThe change goes on next week's read with its judge-by date, so it can never go unjudged.

1Pace

Joins this week, the total against your goal, and the weekly pace needed from here.

readsmet.waitlist_joins_weekly

then Behind pace two weeks running: the constraint review below is mandatory.

2Ladder and constraint

Each rung's count and conversion against its own last 4 weeks, with the weakest rung marked.

readsmet.rung_conversion

then Pick the constraint by the rule: lowest conversion against its own past, counting only rungs with 30 or more entering.

3Where joins came from

Joins by last touch, by first touch, and by what people said when asked.

then A source with joins but no first-touch credit needs a detector feeding it.

4Detector quality

Precision and yield per detector, and the noisiest one.

readsmet.signal_precisionmet.yield_per_dollar

then Retire or retune the worst detector by yield per dollar.

5Draft quality

The share shipped unedited and the median edit distance, split by where the edits fall.

readsmet.drafts_uneditedmet.edit_distance_median

then Edits clustered in the hook: test a new opener prompt.

6Outreach health

Reply rate and stop requests per 100 touches.

readsmet.reply_rate_7dmet.stop_rate

then Stop rate above its guardrail: pause the campaign that caused it.

7Posts

This week's posts, ranked by profile clicks per 100 impressions.

readsmet.impressions_24hmet.profile_clicks_per_100

then Feed the top post's pattern to next week's drafts.

8Experiments due

Each test with a judge-by date this week: arms, sample size reached, and the pre-set rule's verdict.

then Write the verdict to your learnings, win or lose.

9Founder time

Minutes spent per join this week.

readsmet.founder_minutes_per_join

then Rising minutes per join: automate the most time-consuming step first.

10The one change

The constraint chosen, one hypothesis, one change, its judge-by date and its win rule.

0 of 5 filled. Exactly one change. Everything else waits a week.

then Exactly one. Everything else waits a week.

the read, as markdown

# monday read: week of {date}

_fifteen minutes. it ends with one change, and everything else waits a week._

## 1. pace
_Joins this week, the total against your goal, and the weekly pace needed from here._

_nothing written_

then: Behind pace two weeks running: the constraint review below is mandatory.

## 2. ladder and constraint
_Each rung's count and conversion against its own last 4 weeks, with the weakest rung marked._

_nothing written_

then: Pick the constraint by the rule: lowest conversion against its own past, counting only rungs with 30 or more entering.

## 3. where joins came from
_Joins by last touch, by first touch, and by what people said when asked._

_nothing written_

then: A source with joins but no first-touch credit needs a detector feeding it.

## 4. detector quality
_Precision and yield per detector, and the noisiest one._

_nothing written_

then: Retire or retune the worst detector by yield per dollar.

## 5. draft quality
_The share shipped unedited and the median edit distance, split by where the edits fall._

_nothing written_

then: Edits clustered in the hook: test a new opener prompt.

## 6. outreach health
_Reply rate and stop requests per 100 touches._

_nothing written_

then: Stop rate above its guardrail: pause the campaign that caused it.

## 7. posts
_This week's posts, ranked by profile clicks per 100 impressions._

_nothing written_

then: Feed the top post's pattern to next week's drafts.

## 8. experiments due
_Each test with a judge-by date this week: arms, sample size reached, and the pre-set rule's verdict._

_nothing written_

then: Write the verdict to your learnings, win or lose.

## 9. founder time
_Minutes spent per join this week._

_nothing written_

then: Rising minutes per join: automate the most time-consuming step first.

## 10. the one change
- **constraint:** _not set_
- **hypothesis:** _not set_
- **change:** _not set_
- **judge by:** _not set_
- **win rule (set now):** _not set_

then: Exactly one. Everything else waits a week.

how weekly reads go wrong

Too many changessymptomnothing can be judgedfixone change a week; the rest wait
Reading noisesymptoma rung "drops" on 6 peoplefixhide rates below the minimum sample
Vanity firstsymptomimpressions lead the readfixpace and constraint always come first
No readoutsymptomchanges pile up unjudgedfixsection 8 lists them until scored
Moving definitionssymptomnumbers jump when a formula changesfixshow the metric version and compare like with like

Live, daily, weekly

Rhythm

Different questions need different clocks. Live is for what rots, the daily snapshot catches breakage, the weekly read is where decisions are made.

livepush, minutes

A reply or question waiting on you

reads
a reply with no answer
then
answer it; it beats any new prospect
livepush, minutes

A new Hot lead

reads
a lead scored Hot
then
open the lead and draft the move
livepush and email, minutes

A failed payment

reads
a payment failure
then
run the recovery play
livepush, minutes

A stop or report

reads
negative feedback
then
mark never-contact; check the campaign's stop rate
livepush, within the hour

A source down past its max staleness

reads
source health
then
fix the feed or pause its detectors
livepush, minutes

A scheduled post failed

reads
scheduled post status
then
retry or reschedule
dailydashboard, once a day

People on each rung, and new arrivals

reads
the funnel snapshot
then
nothing unless something broke
dailydashboard, once a day

New signals and leads per trigger and detector

reads
signal and lead counts
then
watch for a detector going silent
dailydashboard, once a day

Drafts generated, shipped, edited, discarded

reads
draft decisions
then
a spike in discards means a prompt broke
dailydashboard, once a day

Post metrics from the snapshots

reads
post snapshots
then
feed winners to tomorrow's drafts
weeklyMonday read, Monday

Pace against the goal

reads
joins per week
then
part of the one change
weeklyMonday read, Monday

The constraint review

reads
rung conversion
then
choose one change
weeklyMonday read, Monday

Experiment readouts due

reads
judge-by dates
then
write verdicts to your learnings
monthlymonthly review, first Monday

First-touch attribution

reads
credit by first touch
then
rebalance channels and detectors
monthlymonthly review, first Monday

Label consistency check

reads
a blind relabel of 20 items
then
update the labelling rule
  • Live is for things that rot in hours. Everything else waits for a snapshot.
  • Never decide on live data. A single event is an anecdote; decisions read frozen snapshots.
  • Give every alert a cooldown, so one busy thread does not buzz all afternoon.
  • Count alerts per day against how many you acted on. Ignored alerts mean too many lines.

A line for every number that matters.

Not a dashboard: dashboards wait to be read, thresholds come to you. Thirty-six starting lines, the eight to switch on first, and the arithmetic of alert fatigue. A threshold is a hypothesis too, so every level is a rule of thumb to re-fit from your own data.

threshold starter kit

the first eight to switch on

  1. 1Stop requests per 100 touchesStop requests over 2 per 100 touches auto-pauses a campaign. The one line that protects your name.
  2. 2Holdout leads who received a touchAny touched holdout pauses the campaign. Without it, every lift readout is quietly void.
  3. 3Source freshness (newest item age)A stale source alerts and is marked stalled. Feeds get shut down and change shape without notice: this is the line that notices.
  4. 4Detector precision (founder labels real ÷ labeled)Under 60% over the last 50 labels shadows the detector. It needs labels first, so it switches on once 50 exist.
  5. 5Hot lead with no touchA Hot lead untouched for 48 hours alerts you. The cheapest win on the list.
  6. 6Oldest untouched item in the pain-signal queueThe pain queue's oldest item past 72 hours alerts. Same idea, one step earlier.
  7. 7Joins against the straight line to your goalBehind the straight line to your goal brings the Monday read.
  8. 8Experiment past its judge-by date with no readoutA test past its judge-by date with no readout alerts. Keeps the loop honest from day one.

Every level here is a rule of thumb, not a measured fact. Start silent, label each firing, and re-fit from your own data after the first month.

Phase
Kind of line
Who acts

36 of 36 thresholds

stage 1 · linkwarn

Intake completion rate (people who start the intake, to finished)

type
Change vs baseline
level
Down 30% against its 4-week baseline
window
7 days
min sample
30 intake starts
action
Open hypothesis, owned by you
why
A copy or step change broke the intake. Opens an intake copy or step-order test.
thr.link.intake_drop
stage 1 · linkcritical

Intake step error rate

type
Ceiling
level
Over 5% of runs error on any one step
window
24 hours
min sample
20 runs
action
Alert only, owned by the system
why
Something is broken, not underperforming. Alert first; nothing to test until it's fixed.
thr.link.step_errors
stage 2 · briefwarn

Founder correction rate, per brief field

type
Ceiling
level
Over 25% of one field's values corrected
window
30 days
min sample
20 briefs
action
Queue offline replay, owned by the analyst
why
The extractor is wrong on that field. Replay the next prompt version on every corrected case before shipping it.
thr.brief.field_corrections
stage 2 · briefwarn

Share of 'verified' facts you corrected

type
Ceiling
level
Over 10%
window
30 days
min sample
30 verified facts
action
Open hypothesis, owned by the analyst
why
Verified is supposed to mean quoted and checked. If it is wrong this often, the evidence rules need a test.
thr.brief.tier_a_wrong
stage 2 · briefinfo

Cost per brief run

type
Change vs baseline
level
Up 50% against its 4-week baseline (a cost rise is the bad direction)
window
7 days
min sample
10 runs
action
Alert only, owned by the system
why
Usually a crawl or a prompt grew. Alert only.
thr.brief.cost
stage 3 · icpwarn

Share of reviewed leads the founder marks never

type
Ceiling
level
Over 40%
window
7 days
min sample
30 reviews
action
Open hypothesis, owned by the analyst
why
The filter lets in people the founder won't contact. Test tighter not-if lines by replaying past reviews.
thr.icp.never_rate
stage 3 · icpinfo

Lift of one fit line on reply or payment

type
Floor
level
Leads matching the line reply or pay no more than leads without it
window
90 days
min sample
100 leads on each side
action
Propose retire, owned by you
why
A fit line that predicts nothing is noise. Propose retiring it; the founder approves.
thr.icp.dead_line
stage 4 · momentsinfo

Reply rate on signals older than the trigger's freshness window

type
Floor
level
Under half the reply rate of fresh signals
window
60 days
min sample
40 touches in each age bucket
action
Propose test, owned by the analyst
why
The window is too long. Propose a shorter fresh_for, judged on the next 60 days.
thr.trig.decay
stage 4 · momentswarn

Signals from a live trigger

type
Staleness
level
None in 14 days
window
14 days
min sample
n/a
action
Alert only, owned by the system
why
A live trigger that never fires is either rare or broken. Check its detectors' health.
thr.trig.silent
stage 5 · detectorsswitch on first: 4 of 8warn

Detector precision (founder labels real ÷ labeled)

type
Floor
level
Under 60% over the last 50 founder labels
window
rolling 50 labels
min sample
50 labels
action
Auto-shadow, owned by the system
why
Move the detector to shadow so it stops filling the list, and open a rule test.
thr.det.precision
stage 5 · detectorswarn

Cost per accepted lead, per detector

type
Change vs baseline
level
Up 2× against its 4-week baseline
window
30 days
min sample
20 accepted leads
action
Open hypothesis, owned by the analyst
why
Same leads for more money: vocabulary is too broad or a source got noisy.
thr.det.yield_cost
stage 5 · detectorswarn

Matches per day, per detector

type
Control limit
level
Above the upper control limit (about 3× the 4-week daily median as a first cut)
window
daily
min sample
14 days of history
action
Alert only, owned by the system
why
Spam waves and viral threads look like demand. Alert; shadow it if precision drops too.
thr.det.spike
stage 5 · detectorsswitch on first: 3 of 8warn

Source freshness (newest item age)

type
Staleness
level
Past the source's own max staleness
window
continuous
min sample
n/a
action
Alert only, owned by the system
why
Mark the source's health stalled and alert. A source can be quietly dead while every run reports success.
thr.src.stale
stage 5 · detectorswarn

Share of runs ending partial or error, per source

type
Ceiling
level
Over 10% of runs in 24 hours
window
24 hours
min sample
10 runs
action
Alert only, owned by the system
why
Degraded, not dead. A source that comes back partial on every run is the classic case: this is the line that catches it.
thr.src.errors
stage 6 · signalswarn

Judge and founder disagreement on labeled signals

type
Ceiling
level
Over 20%
window
14 days
min sample
40 labels
action
Queue offline replay, owned by the analyst
why
Replay the next judge prompt on every labeled case; ship only if agreement goes up and recall doesn't drop.
thr.sig.disagree
stage 6 · signalsinfo

Share of matches discarded as no person

type
Change vs baseline
level
Over 2× its 4-week baseline
window
7 days
min sample
100 matches
action
Alert only, owned by the system
why
Usually the identity step broke, or a source changed its author field.
thr.sig.noperson
stage 6 · signalsswitch on first: 6 of 8warn

Oldest untouched item in the pain-signal queue

type
Staleness
level
Over 72 hours
window
continuous
min sample
n/a
action
Alert only, owned by you
why
Signals decay. Tell the founder before the window closes.
thr.sig.queue_age
stage 7 · opportunitieswarn

Reply rate of the daily top 10

type
Control limit
level
Below the lower control limit for 2 weeks running
window
weekly
min sample
30 touches a week
action
Propose test, owned by the analyst
why
The ranking stopped finding responders. Propose a scorer change, replayed on frozen features first.
thr.lead.top10
stage 7 · opportunitieswarn

Share of merges the founder splits

type
Ceiling
level
Over 5%
window
30 days
min sample
40 merges
action
Queue offline replay, owned by the analyst
why
The identity rules are merging strangers. Tighten and replay.
thr.lead.splits
stage 7 · opportunitiesswitch on first: 5 of 8warn

Hot lead with no touch

type
Staleness
level
Over 48 hours
window
continuous
min sample
n/a
action
Alert only, owned by you
why
A hot lead going cold is the most expensive miss in the pipeline.
thr.lead.hot_aging
stage 8 · playswarn

Reply rate per touch, per play

type
Control limit
level
Below the play's lower control limit
window
rolling 4 weeks
min sample
30 touches
action
Propose test, owned by the analyst
why
Open an opener test on that play. Don't change the play and the opener at once.
thr.pb.reply
stage 8 · playsinfo

Share of moves taken from the exploration slice

type
Floor
level
Under 5% of moves
window
7 days
min sample
100 moves
action
Alert only, owned by the system
why
Without exploration, the logged odds stop supporting offline evaluation. Data-quality alert.
thr.pb.explore
stage 8 · playsinfo

Signups from a play

type
Floor
level
Zero after 60 touches
window
since launch
min sample
60 touches
action
Propose retire, owned by you
why
Propose retiring or reworking the play. The founder decides.
thr.pb.dead_play
stage 9 · campaignswitch on first: 1 of 8critical

Stop requests per 100 touches

type
Ceiling
level
Over 2 per 100
window
7 days
min sample
50 touches
action
Auto-pause, owned by the system
why
Pause the campaign now. Reputation damage compounds faster than any test can measure.
thr.camp.stops
stage 9 · campaignswitch on first: 2 of 8critical

Holdout leads who received a touch

type
Ceiling
level
Any
window
continuous
min sample
n/a
action
Auto-pause, owned by the system
why
A touched holdout voids the lift readout. Pause and alert.
thr.camp.holdout
stage 9 · campaigninfo

Campaign past its readout date with no readout

type
Staleness
level
Over 48 hours late
window
continuous
min sample
n/a
action
Alert only, owned by you
why
Unread tests teach nothing. Nudge the readout.
thr.camp.overdue
stage 10 · next moveswarn

Median draft edit distance

type
Change vs baseline
level
Up by a third week over week
window
weekly
min sample
20 decided drafts
action
Queue offline replay, owned by the analyst
why
The drafter drifted from the founder's voice. Replay the current prompt against recent edits to find where.
thr.seq.edit_drift
stage 10 · next moveswarn

Share of drafts shipped unedited

type
Change vs baseline
level
Down 25% against its 4-week baseline
window
weekly
min sample
20 decided drafts
action
Propose test, owned by the analyst
why
Same signal as edit drift, from the other side. Propose a prompt test.
thr.seq.unedited
stage 10 · next moveswarn

Scheduled sends that missed their grace window

type
Ceiling
level
Over 2% of due sends
window
24 hours
min sample
20 due sends
action
Alert only, owned by the system
why
A dispatcher or token problem, not a content problem.
thr.seq.missed
stage 11 · funnelswitch on first: 7 of 8warn

Joins against the straight line to your goal

type
Floor
level
Behind the line
window
weekly
min sample
n/a
action
Monday review, owned by you
why
The Monday review picks the weakest rung as the constraint and opens one hypothesis on it.
thr.fun.pace
stage 11 · funnelwarn

Conversion of one funnel rung

type
Control limit
level
Below its lower control limit 2 weeks running
window
weekly
min sample
30 entering the rung
action
Open hypothesis, owned by the analyst
why
That rung is the constraint. Open one hypothesis on it, not three.
thr.fun.rung
stage 11 · funnelwarn

Stripe invoice.payment_failed

type
Event
level
Any
window
event
min sample
n/a
action
Start a play, owned by the system
why
Not a threshold so much as a warm trigger: start the failed-payment recovery sequence.
thr.fun.payfail
stage 11 · funnelinfo

A post's 30-minute impressions

type
Floor
level
Below the account's 10th percentile at 30 minutes
window
per post
min sample
the account's post history
action
Log only, owned by the system
why
Logged for the early-read forecast. No action: a single post is noise.
thr.fun.early
stage 12 · loopswitch on first: 8 of 8warn

Experiment past its judge-by date with no readout

type
Staleness
level
Over 48 hours
window
continuous
min sample
n/a
action
Alert only, owned by you
why
Every test gets read, win or lose.
thr.loop.overdue
stage 12 · loopinfo

Share of a threshold's firings labeled not real

type
Ceiling
level
Over 50% of its last 10 firings
window
last 10 firings
min sample
10 firings
action
Re-fit threshold, owned by the analyst
why
The threshold cries wolf. Re-fit its level from the last 8 weeks and version the change.
thr.loop.falsealarm
stage 12 · loopinfo

Founder minutes spent on the pipeline

type
Ceiling
level
Over the week's stated budget
window
weekly
min sample
n/a
action
Alert only, owned by you
why
A guardrail on the founder's time: the pipeline should save hours, not eat them.
thr.loop.minutes

alert fatigue: many lines, checked often, will fire by chance

0.27%chance per check
140checks a week
32%weeks with at least one false alarm
0.38expected false alarms a week

Chance per check is 2(1 minus the normal cdf at the width). Chance of any is 1 minus (1 minus that) to the number of checks. Twenty lines checked daily at 3 sigma is roughly a 31% chance of a false alarm every week, which is why fewer, sharper lines beat a wall of alerts.

six rules for tuning the lines themselves

  1. 1Start from a baselineBefore a line goes live, compute its metric over the last 4 to 8 weeks. Start the level at the rule of thumb or at the baseline's control limit, whichever is wider.
  2. 2Run it silent firstNew lines fire as log only for 2 weeks. Count how often each would have fired.
  3. 3Label every firingEach firing gets real, false alarm or unsure. The readout of any test it opened settles the label.
  4. 4Re-fit monthlyRecompute the baseline and limits on the current version of whatever it watches. More than half false alarms widens the line; a labelled miss tightens it.
  5. 5Version every changeA new level is a new version with a reason. Old firings stay tied to the version that fired.
  6. 6Retire dead linesA line that has not fired in 90 days and watches nothing critical becomes log only. Fewer, sharper lines beat a wall of alerts.

how a line fires, and the seven kinds

  1. 1Read the metricover its window, from a frozen snapshot
  2. 2Sample gateenough units? if not, stay silent
  3. 3Comparethe value against the level or control limits
  4. 4Cooldown and mutefired recently? muted? then stop
  5. 5Firewrite the alert and run its action
Floor
A rate must stay above a level: detector precision, play sign-ups.
Ceiling
A rate must stay below a level: stop requests, correction rate, never rate.
Control limit
Fires when a point falls outside limits drawn from the metric's own history: reply rate per play, rung conversion.
Change vs baseline
Fires on a relative move against a rolling baseline: cost per lead up 2x, edit distance up a third.
Sample-size gate
Not a line by itself: the minimum units before any other line may fire. Every row has one.
Staleness
Fires when something has not happened for too long: source freshness, a Hot lead untouched, an overdue readout.
Event
A single event that always matters: a failed payment starts a recovery play.

Came after, not caused.

Twelve kinds of touch, ranked by how much evidence they carry back to a sign-up. Credit models tell you what came before; only a holdout tells you what a campaign caused.

attribution
Keep every model. Last touch, first touch and self-reported, side by side. Never pick one and throw the rest away.
Count the orphans. Sign-ups with no link and no answer are a number, not a shrug.
Cause needs a holdout. Never say a campaign "caused" sign-ups without one. Say "came after".
How traceable

Each kind of touch carries a different amount of evidence back to the sign-up. Fully traceable ones join a click to a person; the rest rest on what people tell you.

X, Threads, Redditfully traceable

Public reply with a link in a reply

carries
a short link with a message id
measure
clicks and visits; replies by thread
note
high once every message carries an id
X, Instagramfully traceable

DM with a short link

carries
a short link with a message id
measure
click, visit, sign-up
X, Instagrammostly self-reported

DM with no link

carries
nothing travels
measure
only the reply, and the self-reported source
emailfully traceable

Email

carries
a tracked link and a message id
measure
click, visit, sign-up
X, Threadspartly traceable

Own post, link in reply

carries
a short link per post
measure
clicks per post; profile clicks
X, Threads, TikTok, Reelsmostly self-reported

Own post, no link

carries
none (view-through)
measure
profile clicks, follows, self-reported
X, Instagramfully traceable

Comment-a-keyword reply

carries
an automatic DM with a short link
measure
keyword comments, clicks
every platformpartly traceable

Waitlist page from a bio link

carries
a bio short link
measure
visits from the bio link
note
no message-level credit
search, AI answerspartly traceable

Comparison or alternatives page

carries
a UTM on internal links; the referrer
measure
landing page and referrer
ChatGPT, Claude, Perplexitymostly self-reported

Named by an AI assistant

carries
a referrer when linked; often none
measure
self-reported; share of answers that name you
Product Hunt, Uneed, BetaListpartly traceable

Directory or launch site

carries
referrer and UTM
measure
visits and sign-ups by referrer
anywheremostly self-reported

Word of mouth

carries
nothing
measure
self-reported only

four ways to give credit

Last touch
All credit to the final touch before the sign-up. It rewards the step that closed.
First touch
All credit to the touch that opened the door. It tells you which triggers find people.
Self-reported
"How did you hear about us?" in the person's own words. It catches what links miss: a friend, a podcast, a post they saw but did not click.
Holdout lift
What happened because of the touch, against a group that did not get it. The only causal one.

the short-link and "how did you hear" checklist

0 of 8 in place. A DM that was read but not clicked leaves no trail; the question on the form is how you catch it.

does a channel find people, or only close them?

public replies: Both finds and closes.

DMs: Closes but does not find. Something upstream has to feed it people.

own posts: Both finds and closes.

Example numbers. A channel with lots of last-touch credit and none first-touch closes but does not find. The reverse finds but does not close.

came after, not caused

Say it came after.

Sign-ups came after the reply campaign. Without a holdout, that is all you can say.

When a line fires, do this.

Ten runbooks, one per kind of firing. Contain, diagnose, fix or test, approve, close. Contain first, then understand, then change one thing.

runbooks
  1. 1contain
  2. 2diagnose
  3. 3fix or test
  4. 4approve
  5. 5close

Contain first, then understand, then change one thing. It can contain and it can propose. It never ships a live change without you: every idea is a row you can approve in one tap. A firing with no runbook becomes a notification someone ignores.

Severity

10 of 10 runbooks

Detector qualitywarnfires on: Precision under its floor, or a match spike past the control limit
  1. systemMove the detector to shadow. It keeps logging, stops filling the list.
  2. analystPull 20 recent matches; mark which vocabulary terms or rule fired on the ones labeled not real.
  3. analystDraft vocab or rule v-next and queue an offline replay on stored matches with labels.
  4. youApprove go-live if precision clears the floor and labeled-real matches aren't lost.
test it opens
Offline replay, then shadow beside live for a week
closes when
Precision back above the floor on 50 new labels, or the detector is retired
logs
alert, replay id, old and new version, precision and recall on the replay, decision
Source stale or failingwarnfires on: Newest item older than max staleness, or over 10% of runs partial or error
  1. systemMark the source stalled or degraded; alert. Detectors on it are marked as missing input, so their silence isn't read as no demand.
  2. analystCheck the last good run, the error text and whether the feed URL or API changed.
  3. youDecide: fix, swap to another source, or retire. A swap keeps the same row shape.
test it opens
None for breakage; a replacement source runs in shadow until its matches look like the old one's
closes when
Fresh items flowing for 24 hours, or source retired
logs
alert, error class, downtime, items missed estimate, replacement source id
Reputation guardrailcriticalfires on: Stop requests over 2 per 100 touches, a touched holdout, or a spam report
  1. systemPause the campaign immediately.
  2. analystList the touches that drew stops: play, step, template, channel, time.
  3. youRead them. Decide: kill the step, rewrite the template, or narrow the segment.
  4. youApprove a restart only with the change in place.
test it opens
The rewritten step restarts as a new variant against the old (if kept) with a fresh stop-rate guardrail
closes when
Campaign resumed with the change, or ended
logs
alert, touches involved, template versions, decision, restart date
Play underperformingwarnfires on: Reply rate per touch below the play's lower control limit (4-week window)
  1. analystSplit the drop by step, channel, lead tier and trigger age. Rule out a source or lead-mix change first.
  2. analystIf the mix is stable, draft an opener test: current opener vs one challenger, same play.
  3. youApprove the test; it runs with a holdout share and a pre-set win rule.
test it opens
Randomized opener test on the play's next touches
closes when
Readout done and decision logged
logs
alert, diagnosis notes, hypothesis id, readout id
Draft driftwarnfires on: Median edit distance up a third week over week, or unedited share down 25%
  1. analystGroup the week's edits by edit locus (hook, body, ending) to see where the founder rewrites.
  2. analystQueue an offline replay: current prompt vs a v-next tuned on the edits, scored against what was actually posted.
  3. youApprove v-next if the replay brings drafts closer to what was posted.
test it opens
Offline replay against posted text, then a live slot test via the randomized draft slot
closes when
Edit distance back within its limits for 2 weeks
logs
alert, edit locus breakdown, prompt versions, replay scores
Lead ranking offwarnfires on: Top-10 reply rate below its lower control limit 2 weeks running, or founder splits over 5% of merges
  1. analystReplay the scorer on frozen features: does a reweighted version rank past repliers higher?
  2. analystFor merges: list the split pairs and the rule that joined them; tighten and replay.
  3. youApprove the new scorer or merge rule.
test it opens
Offline replay first; then a randomized slice of the daily list ranked by the new scorer
closes when
Readout done
logs
alert, replay metrics, scorer versions
Funnel rung below limitswarnfires on: One rung's conversion below its lower control limit 2 weeks running
  1. analystConfirm the rung's input volume and mix didn't change; find the step that feeds it.
  2. youMonday review: make it the constraint for the week.
  3. analystDraft one hypothesis aimed at that rung.
test it opens
One randomized test on the rung (offer, copy or step order)
closes when
Rung back inside limits, or test read out
logs
alert, constraint choice, hypothesis id
Behind the goal pacewarnfires on: Waitlist joins behind the straight line to the goal
  1. systemPut the pace line and the rung chart at the top of the Monday review.
  2. youPick the weakest rung as the week's constraint.
  3. analystOpen one hypothesis on that constraint; park other open ideas.
test it opens
Whatever the constraint's hypothesis needs
closes when
The week's review is done
logs
pace, constraint chosen, hypothesis id
Readout or label overdueinfofires on: A test past its judge-by date, a campaign past its readout date, or a firing unlabeled after 7 days
  1. systemNudge the owner with the one action needed.
  2. analystPre-compute the readout so it only needs a decision.
test it opens
None
closes when
Readout written or label set
logs
overdue item, days late
Threshold crying wolfinfofires on: Over half of a threshold's last 10 firings labeled false alarm
  1. analystRe-fit the level from the last 8 clean weeks; draft it as a new version.
  2. analystRun the new level silent beside the old for 2 weeks.
  3. youApprove the swap if it would have caught the real firings with fewer false ones.
test it opens
Silent side-by-side of two threshold versions
closes when
New version live, or old kept with a reason
logs
old and new levels, back-test firings vs labels

How this works.

how this works

The sizing and the significance checks use the standard two-proportion formulas at a 5% two-sided level:

n per arm ≈ 16 × p̄(1 − p̄) / d²   (80% power)
z = (p̂₁ − p̂₀) / √(p̄(1 − p̄)(1/n₀ + 1/n₁))
p-chart sigma = √(p̄(1 − p̄) / n), act at 3 sigma
the lower limit clears zero only when n > 9(1 − p̄) / p̄

Fisher's exact test takes over when either arm has fewer than five successes or five misses. The Bayesian read puts a Beta prior on each rate and reports the chance the variant is better, a 90% range and the expected loss. The simulator is seeded, so everyone sees the same numbers for the same settings, and it runs in small slices so the page stays responsive.

This page is the method, to run by hand or in a spreadsheet. Every baseline, threshold and kill line is a rule of thumb, re-fit from your own data after the first month. A small account can only detect big effects, so test big changes and write the sample size down first. Nothing here means a tool runs your tests, writes your Monday read or sends your alerts for you.

want this run for you? vibedraft finds the people ready to buy on X and in your own signups, drafts the right move in your voice, keeps your channels posting and learns what pays. you approve everything that goes out.

Start here

Runs in your browser. Nothing you type here is sent anywhere.