all guides

The improvement agent

An agent that diagnoses, proposes and queues the next test.

learn · improvement loop

An agent that does the analyst's job every morning. It reads the daily snapshot and the night's firings, digs into the data behind each one, and writes the next hypothesis with the effect it expects. Then it queues an offline replay or drafts a test through the hypothesis registry. It can contain and it can propose. It never ships a live change without the founder.

snapshot + firings → agent · diagnose, propose, queue → registry → founder approves → engine runs

An analyst agent with a narrow, auditable job.

what it is, in industry terms

Product teams call this person the growth analyst: they watch the dashboards, find what moved, and bring a test to the weekly review. Here an agent does that part on a schedule, inside guardrails. In autoresearch terms it's a ratchet loop: propose a change, evaluate it against a fixed rule, keep wins, discard losses. It works on the pipeline's own components instead of a single prompt.

it does

  • Read. The daily snapshot, open firings, recent readouts and the learnings base.
  • Diagnose. Slice the metric behind a firing until one part explains it.
  • Propose. One hypothesis per firing, with expected effect, unit, metric and the cheapest test that can settle it.
  • Queue. Offline replays and shadow runs it can start alone; live tests as drafts for the founder.
  • Tidy. Re-fit noisy thresholds, chase overdue readouts, label obvious false alarms for the founder to confirm.

it doesn't

  • Ship to people. No live template, offer, segment or play change without the founder.
  • Stack changes. One open test per knob; new ideas wait in the queue.
  • Grade its own work. Readouts use the rule set at registration, computed in code, not by the agent.
  • Edit history. It can't change a baseline, a label or a readout after the fact; only add new versions.

One run a morning, plus a Monday brief.

how it works

  1. Read — snapshot, firings learnings
  2. Triage — real? already being handled?
  3. Diagnose — slice until one part explains it
  4. Propose — one hypothesis + expected effect
  5. Queue — replay now or draft a test
  6. Brief — founder approves in one tap
  1. Read — Load the daily snapshot, every unlabeled firing, open tests, recent readouts and the learnings for the scopes involved.

  2. Triage — Drop firings already covered by an open test or runbook. Flag likely false alarms (tiny n, known incident) for the founder to confirm.

  3. Diagnose — For each remaining firing, slice the metric by version, source, play, step, tier, weekday. Stop when one slice explains most of the move, or say it can't tell.

  4. Propose — Write one hypothesis per real problem: the change, the mechanism, the expected effect with a range, the unit, the metric, the test mode, and the learnings it leans on.

  5. Queue — Start offline replays and shadow runs directly. Draft live tests with arms, sample size and win rule, and leave them for approval.

  6. Brief — A short note for the founder: what fired, what it thinks, what it already started, what needs a yes. Mondays, it also proposes the week's constraint.

Four levels of autonomy.

what it may do

level 0 — Observe · Reads and writes notes only. Use it for the first 2 weeks to check its diagnoses against the founder's.

level 1 — Contain and replay · May auto-shadow, mark sources stalled, queue offline replays, and re-fit thresholds as silent versions.

level 2 — Draft tests · May write hypotheses and full test drafts into the registry. Nothing live starts without approval.

level 3 — Run approved classes · May start pre-approved test classes on its own, e.g. offline-validated detector vocabulary changes. Never anything that changes a message, offer or segment.

Every idea is a row the founder can approve in one tap.

the proposal object

FieldTypeWhat it holds
proposal_iduuidOne row per idea
from_alertsalert idsThe firings that led here
diagnosistext + evidenceWhat it found, with the slices and numbers behind it
hypothesistextIf we change X, metric Y moves by Z, because M
expected_effectrangeThe effect it predicts, written before any test
knob + versionrefExactly what would change
modeenumoffline replay · shadow · A/B · holdout · off-policy
test_draftrefA full draft in the hypothesis registry: unit, arms, sample size, win rule, judge-by date
costobjectCredits, founder minutes, days to a readout
priorslearning idsPast learnings it relied on
autonomylevelWhat it may do with this on its own
statusenumqueued · needs approval · approved · running · read out · rejected (with reason)
outcomereadout idFilled when the test reads out; scores the proposal

a proposal after an edit-drift firing

{
 "proposal_id": "prop_3c1e…",
 "from_alerts": [
 "al_9d02…"
 ],
 "diagnosis": {
 "finding": "most rewrites are in the hook of reply drafts",
 "slices_checked": [
 "register",
 "play",
 "weekday",
 "prompt_version"
 ]
 },
 "hypothesis": "Leading with the lead's own words cuts hook rewrites, because the founder rewrites generic openers",
 "expected_effect": {
 "metric": "median edit distance",
 "change": "-15% to -30%"
 },
 "knob": {
 "name": "reply opener prompt",
 "from": "v5",
 "to": "v6"
 },
 "mode": "offline replay, then randomized slot",
 "test_draft": "hyp_seq_opener_v6",
 "cost": {
 "credits": "replay only",
 "founder_minutes": 5,
 "days_to_readout": 14
 },
 "priors": [],
 "autonomy": 2,
 "status": "needs approval",
 "outcome": null,
 "note": "illustration"
}

A small toolbox, each tool scoped.

its tools

read_snapshot

does
Daily counts per rung, trigger, detector, play, campaign.
needs
read only

read_alerts

does
Open and recent firings with frozen values and baselines.
needs
read only

slice_metric

does
Recompute one metric split by a dimension, on frozen rows.
needs
read only

read_learnings

does
Past readouts and effect sizes for a scope or knob.
needs
read only

queue_replay

does
Run a version of a prompt, rule or scorer on stored items and labels.
needs
level 1

set_shadow

does
Move a detector or lane to shadow.
needs
level 1

refit_threshold

does
Write a new threshold version that runs silent first.
needs
level 1

draft_hypothesis

does
Write a hypothesis and test draft into the registry.
needs
level 2

request_approval

does
Put a proposal in the founder's queue with a one-line ask.
needs
level 2

label_suggest

does
Suggest real or false alarm on a firing; the founder confirms.
needs
level 0

Judge the agent by its proposals' outcomes.

its own eval

The agent is a component like any other, so it gets metrics and thresholds too. Most of them can only be read once readouts come back, which is why every proposal stores its expected effect up front.

MetricDefinitionWhy it matters
Proposal hit rateShare of read-out proposals whose test was a win by its pre-set ruleIs it finding real problems and plausible fixes?
Effect calibrationHow often the measured effect lands inside the expected rangeDoes it know how big its ideas are? Feeds the founder's trust.
Diagnosis agreementShare of diagnoses the founder agrees with at level 0The first gate before giving it more autonomy.
Approval rateShare of proposals the founder approvesToo low means noise; very high may mean timid.
Time to proposalHours from a firing to a proposal in the queueThe loop's speed.
Win sizeMean effect of winning proposals on the constraintGuards against many tiny safe wins.
Harm rateProposals whose test tripped a guardrailShould be near zero; any case is reviewed.

keep it honest

Proposal hit rate is easy to game by proposing safe, tiny changes. Read it beside the mean effect of winners and the share of proposals the founder approves. A good agent wins less often than a timid one but moves the constraint more.

A Monday brief, as the agent would write it.

illustrative example

The firings below are an illustration of the format, not measurements.

  • Fired overnight (illustration). Detector precision on one Reddit pain detector fell under its floor; I moved it to shadow and queued a vocabulary replay. Readout tomorrow.
  • Needs your yes. Test: offer the brief in the first reply to Hot leads vs the current reply. Unit: lead. Holdout 10%. Primary metric: brief offered → waitlist offered. Judge-by in 3 weeks.
  • Parked. Opener rewrite on reply drafts: it waits until the brief-offer test reads out, since both touch the first reply.
  • Housekeeping. Two firings look like false alarms (under the minimum sample before a source outage); please confirm.

How an improvement agent goes wrong.

failure modes

Story-telling — The agent explains every wiggle with a confident story. · It must name the slices it checked and say 'can't tell' when no slice explains the move; diagnosis agreement is tracked.

Proposal spam — Ten ideas a day; the founder stops reading. · One proposal per real problem, one live test per knob, a daily cap on the queue.

Goodharting its eval — Proposes only safe changes to keep its hit rate high. · Read hit rate beside win size and approval rate.

Leaking the future — Diagnoses using data recomputed after the fact. · Its tools read frozen rows only: features, baselines and alerts as stored.

Autonomy creep — A convenient shortcut lets it change a live template. · Tools are scoped by level; nothing that changes what a person sees has a write tool.

The agent's runs are training data for the next agent.

data to capture

record — Every run: inputs read, firings triaged, slices computed, proposals written, actions queued, tokens and cost.

label — Founder agreement with each diagnosis; each proposal's readout and whether its effect landed in the expected range.

trains — A better proposal ranker and diagnosis prompts, evaluated on past runs before each prompt change, the same way drafts are.

test — New agent prompt versions run in shadow on the same mornings and are scored against the live agent's later readouts.

More in Improvement loop