learn · improvement loop
The improvement agent
An agent that diagnoses, proposes and queues the next test.
In Improvement loop
An agent that does the analyst's job every morning. It reads the daily snapshot and the night's firings, digs into the data behind each one, and writes the next hypothesis with the effect it expects. Then it queues an offline replay or drafts a test through the hypothesis registry. It can contain and it can propose. It never ships a live change without the founder.
snapshot + firings → agent · diagnose, propose, queue → registry → founder approves → engine runs
what it is, in industry terms
An analyst agent with a narrow, auditable job.
Product teams call this person the growth analyst: they watch the dashboards, find what moved, and bring a test to the weekly review. Here an agent does that part on a schedule, inside guardrails. In autoresearch terms it's a ratchet loop: propose a change, evaluate it against a fixed rule, keep wins, discard losses. It works on the pipeline's own components instead of a single prompt.
it does
- Read. The daily snapshot, open firings, recent readouts and the learnings base.
- Diagnose. Slice the metric behind a firing until one part explains it.
- Propose. One hypothesis per firing, with expected effect, unit, metric and the cheapest test that can settle it.
- Queue. Offline replays and shadow runs it can start alone; live tests as drafts for the founder.
- Tidy. Re-fit noisy thresholds, chase overdue readouts, label obvious false alarms for the founder to confirm.
it doesn't
- Ship to people. No live template, offer, segment or play change without the founder.
- Stack changes. One open test per knob; new ideas wait in the queue.
- Grade its own work. Readouts use the rule set at registration, computed in code, not by the agent.
- Edit history. It can't change a baseline, a label or a readout after the fact; only add new versions.
how it works
One run a morning, plus a Monday brief.
- Read — snapshot, firings learnings
- Triage — real? already being handled?
- Diagnose — slice until one part explains it
- Propose — one hypothesis + expected effect
- Queue — replay now or draft a test
- Brief — founder approves in one tap
Read — Load the daily snapshot, every unlabeled firing, open tests, recent readouts and the learnings for the scopes involved.
Triage — Drop firings already covered by an open test or runbook. Flag likely false alarms (tiny n, known incident) for the founder to confirm.
Diagnose — For each remaining firing, slice the metric by version, source, play, step, tier, weekday. Stop when one slice explains most of the move, or say it can't tell.
Propose — Write one hypothesis per real problem: the change, the mechanism, the expected effect with a range, the unit, the metric, the test mode, and the learnings it leans on.
Queue — Start offline replays and shadow runs directly. Draft live tests with arms, sample size and win rule, and leave them for approval.
Brief — A short note for the founder: what fired, what it thinks, what it already started, what needs a yes. Mondays, it also proposes the week's constraint.
what it may do
Four levels of autonomy.
level 0 — Observe · Reads and writes notes only. Use it for the first 2 weeks to check its diagnoses against the founder's.
level 1 — Contain and replay · May auto-shadow, mark sources stalled, queue offline replays, and re-fit thresholds as silent versions.
level 2 — Draft tests · May write hypotheses and full test drafts into the registry. Nothing live starts without approval.
level 3 — Run approved classes · May start pre-approved test classes on its own, e.g. offline-validated detector vocabulary changes. Never anything that changes a message, offer or segment.
the proposal object
Every idea is a row the founder can approve in one tap.
| Field | Type | What it holds |
|---|---|---|
proposal_id | uuid | One row per idea |
from_alerts | alert ids | The firings that led here |
diagnosis | text + evidence | What it found, with the slices and numbers behind it |
hypothesis | text | If we change X, metric Y moves by Z, because M |
expected_effect | range | The effect it predicts, written before any test |
knob + version | ref | Exactly what would change |
mode | enum | offline replay · shadow · A/B · holdout · off-policy |
test_draft | ref | A full draft in the hypothesis registry: unit, arms, sample size, win rule, judge-by date |
cost | object | Credits, founder minutes, days to a readout |
priors | learning ids | Past learnings it relied on |
autonomy | level | What it may do with this on its own |
status | enum | queued · needs approval · approved · running · read out · rejected (with reason) |
outcome | readout id | Filled when the test reads out; scores the proposal |
a proposal after an edit-drift firing
{
"proposal_id": "prop_3c1e…",
"from_alerts": [
"al_9d02…"
],
"diagnosis": {
"finding": "most rewrites are in the hook of reply drafts",
"slices_checked": [
"register",
"play",
"weekday",
"prompt_version"
]
},
"hypothesis": "Leading with the lead's own words cuts hook rewrites, because the founder rewrites generic openers",
"expected_effect": {
"metric": "median edit distance",
"change": "-15% to -30%"
},
"knob": {
"name": "reply opener prompt",
"from": "v5",
"to": "v6"
},
"mode": "offline replay, then randomized slot",
"test_draft": "hyp_seq_opener_v6",
"cost": {
"credits": "replay only",
"founder_minutes": 5,
"days_to_readout": 14
},
"priors": [],
"autonomy": 2,
"status": "needs approval",
"outcome": null,
"note": "illustration"
}its tools
A small toolbox, each tool scoped.
read_snapshot
- does
- Daily counts per rung, trigger, detector, play, campaign.
- needs
- read only
read_alerts
- does
- Open and recent firings with frozen values and baselines.
- needs
- read only
slice_metric
- does
- Recompute one metric split by a dimension, on frozen rows.
- needs
- read only
read_learnings
- does
- Past readouts and effect sizes for a scope or knob.
- needs
- read only
queue_replay
- does
- Run a version of a prompt, rule or scorer on stored items and labels.
- needs
- level 1
set_shadow
- does
- Move a detector or lane to shadow.
- needs
- level 1
refit_threshold
- does
- Write a new threshold version that runs silent first.
- needs
- level 1
draft_hypothesis
- does
- Write a hypothesis and test draft into the registry.
- needs
- level 2
request_approval
- does
- Put a proposal in the founder's queue with a one-line ask.
- needs
- level 2
label_suggest
- does
- Suggest real or false alarm on a firing; the founder confirms.
- needs
- level 0
its own eval
Judge the agent by its proposals' outcomes.
The agent is a component like any other, so it gets metrics and thresholds too. Most of them can only be read once readouts come back, which is why every proposal stores its expected effect up front.
| Metric | Definition | Why it matters |
|---|---|---|
| Proposal hit rate | Share of read-out proposals whose test was a win by its pre-set rule | Is it finding real problems and plausible fixes? |
| Effect calibration | How often the measured effect lands inside the expected range | Does it know how big its ideas are? Feeds the founder's trust. |
| Diagnosis agreement | Share of diagnoses the founder agrees with at level 0 | The first gate before giving it more autonomy. |
| Approval rate | Share of proposals the founder approves | Too low means noise; very high may mean timid. |
| Time to proposal | Hours from a firing to a proposal in the queue | The loop's speed. |
| Win size | Mean effect of winning proposals on the constraint | Guards against many tiny safe wins. |
| Harm rate | Proposals whose test tripped a guardrail | Should be near zero; any case is reviewed. |
keep it honest
Proposal hit rate is easy to game by proposing safe, tiny changes. Read it beside the mean effect of winners and the share of proposals the founder approves. A good agent wins less often than a timid one but moves the constraint more.
illustrative example
A Monday brief, as the agent would write it.
The firings below are an illustration of the format, not measurements.
- Fired overnight (illustration). Detector precision on one Reddit pain detector fell under its floor; I moved it to shadow and queued a vocabulary replay. Readout tomorrow.
- Needs your yes. Test: offer the brief in the first reply to Hot leads vs the current reply. Unit: lead. Holdout 10%. Primary metric: brief offered → waitlist offered. Judge-by in 3 weeks.
- Parked. Opener rewrite on reply drafts: it waits until the brief-offer test reads out, since both touch the first reply.
- Housekeeping. Two firings look like false alarms (under the minimum sample before a source outage); please confirm.
failure modes
How an improvement agent goes wrong.
Story-telling — The agent explains every wiggle with a confident story. · It must name the slices it checked and say 'can't tell' when no slice explains the move; diagnosis agreement is tracked.
Proposal spam — Ten ideas a day; the founder stops reading. · One proposal per real problem, one live test per knob, a daily cap on the queue.
Goodharting its eval — Proposes only safe changes to keep its hit rate high. · Read hit rate beside win size and approval rate.
Leaking the future — Diagnoses using data recomputed after the fact. · Its tools read frozen rows only: features, baselines and alerts as stored.
Autonomy creep — A convenient shortcut lets it change a live template. · Tools are scoped by level; nothing that changes what a person sees has a write tool.
data to capture
The agent's runs are training data for the next agent.
record — Every run: inputs read, firings triaged, slices computed, proposals written, actions queued, tokens and cost.
label — Founder agreement with each diagnosis; each proposal's readout and whether its effect landed in the expected range.
trains — A better proposal ranker and diagnosis prompts, evaluated on past runs before each prompt change, the same way drafts are.
test — New agent prompt versions run in shadow on the same mornings and are scored against the live agent's later readouts.