learn · experimentation engine

Off-policy evaluation

Judging a new policy on logged decisions.

In Experimentation engine

The next-best-action engine picks one move per lead: reply in public, DM, email, wait. A smarter picker could reply more often, but testing it live means trying worse moves on real people. Off-policy evaluation estimates how a new picker would have done, using only the old picker's logged decisions, as long as each decision was logged with the probability it had of being chosen. That probability is the whole trick, and most systems never record it.

context → logged move + its odds → reward → estimate a new policy offline → small live slice → ship

what it is, in industry terms

Counterfactual evaluation from logged bandit feedback.

Recommendation, ads and search teams use off-policy evaluation (OPE) to judge a new ranking or bidding policy before it serves anyone. The setting is called a contextual bandit. For each unit you see a context, take one action, and observe a reward for that action only. You never see what the other actions would have earned. OPE re-weights the logged rewards so they speak for a different policy.

Context — x · What the picker knew: lead type, trigger, freshness, fit lines, channel, past touches.

Action — a · The move it chose: a play, a channel, an opener, or wait.

Reward — r · The outcome in a fixed window: replied within 7 days, signed up within 14.

Logging policy — π₀ · behavior policy · The picker that actually ran, with the probability it gave each action.

Target policy — π · candidate · The new picker you want to judge without running it.

Propensity — π₀(a | x) · The probability the logging policy gave the action it took. Logged at decision time.

Importance weight — w = π(a|x) / π₀(a|x) · How much more or less the new policy would have taken this action.

Support — overlap · Every action the new policy takes must have had a nonzero chance under the old one.

how it works

Log the odds at decision time, re-weight later.

Each move is logged with the context the picker saw, every candidate it considered with its score, the action chosen and its probability. When the reward window closes, the reward is joined on the move id. A candidate policy is then scored against the same rows: rows where it agrees with what happened count for more or less, depending on how likely that action was. Rows where it would have done something else count for nothing.

Context x — lead type, trigger, freshness, fit, history
Allowed — set contact policy removes blocked moves
Old policy — π₀ picks a with probability π₀(a∣x), logged
Log the — move row before anything is sent
Reward r — replied within 7 days, joined on move id
Score new — policy π weight w = π / π₀ IPS · SNIPS · DR
Live slice — if it wins randomized, with its own logged odds

the estimators

Four ways to turn logs into an estimate.

IPS V̂ = (1/n) Σᵢ wᵢ · rᵢ wᵢ = π(aᵢ|xᵢ) / π₀(aᵢ|xᵢ) SNIPS V̂ = Σᵢ wᵢ · rᵢ / Σᵢ wᵢ self-normalized: lower variance, slight bias DM V̂ = (1/n) Σᵢ Σₐ π(a|xᵢ) · r̂(xᵢ, a) a reward model r̂ fills in every action DR V̂ = DM + (1/n) Σᵢ wᵢ · (rᵢ − r̂(xᵢ, aᵢ)) doubly robust: right if either π₀ or r̂ is right ESS = (Σ wᵢ)² / Σ wᵢ² effective sample size: how many rows really count

Estimator	Bias	Variance	Needs	Use it when
IPS	none, if propensities are right	high, explodes with small propensities	logged propensities	many rows and moderate weights
SNIPS	small, shrinks with more rows	lower than IPS	logged propensities	the default for small logs
Direct method (DM)	as large as the reward model's error	low	a reward model r̂(x, a)	as a sanity check, or with no exploration (and then distrust it)
Doubly robust (DR)	none if either the propensities or r̂ is right	usually lowest	both	the default once a reward model exists

worked example

Judge a new playbook picker on logs from the old one.

A toy version of a playbook picker's choice: three kinds of lead, three plays. The old picker DMs everyone, except for an exploration slice where it picks uniformly at random. The new picker sends each lead type its best play. Draw logs from the old picker, estimates the new picker's reply rate four ways, and repeats 200 times to show how much each estimate wobbles. Reply rates are invented for illustration.

logging setup

An exploration slice ε of 10% (a uniform random play) and 1,000 logged moves.

Lead type · share	public reply	DM	email
cold pain post · 50%	12%	5%	2%
warm signup, stalled · 30%	4%	10%	18%
competitor-unhappy · 20%	8%	15%	6%

True reply rates per play (illustrative). Bold is the new picker's choice. The old picker DMs every type.

estimated reply rate of the new picker · 200 repeats

14.4% — true reply rate of the new picker

8.5% — reply rate the old picker got

41 — effective sample size of 1,000 logs

IPS — 14.7%

SNIPS — 14.7%

DM — 14.7%

DR — 14.7%

Each figure is the average estimate across the 200 repeats; all four land close to the true 14.4%, and the effective sample size of 41 says how few of the 1,000 logs actually inform the new picker.

with ε = 0

With no exploration, the old picker never sends a public reply or an email, so the logs say nothing about the new picker's main moves. IPS and SNIPS collapse, and the direct method can only guess. That's the support problem, and no estimator fixes it after the fact.

support and overlap

You can only evaluate moves the old policy sometimes made.

Deterministic logging kills OPE. A picker that always takes its top move logs a propensity of 1 for it and 0 for everything else. Any new policy that differs anywhere has no data to stand on.
Small propensities explode variance. A move taken with 2% probability gets a weight of 50. A handful of such rows decide the estimate. Watch the effective sample size, and clip weights (cap w at, say, 20) with the bias that brings.
Propensities must be logged, not reconstructed. Recomputing "what the probability would have been" later, from a changed scorer, is guessing. Write it at decision time with the policy version.

Context must be frozen. If the context is recomputed later, the new policy is scored on information it wouldn't have had (rule 3 again).
Rewards need a fixed window. "Replied within 7 days" must be closed before a row counts, or recent moves look worse than old ones.
Blocked moves aren't actions. If the contact policy forbade a move for this lead, neither policy could take it. Log the allowed set, and explore only inside it.

exploration

Pay a little now to be able to learn later.

Exploration means sometimes taking a move the picker doesn't rank first, on purpose and at a logged probability. It costs some replies today. It buys the ability to evaluate any future picker offline, and to learn moves nobody has tried. The safety rule: explore only among moves the contact policy allows for that lead, and never on leads marked never-contact.

Strategy	How it picks	Propensity	Cost	Fit for a small founder
Uniform slice (ε-greedy)	with probability ε pick any allowed move at random; otherwise the top move	exact: 1 − ε + ε/k for the top move, ε/k for each other	about ε of moves are not the best guess	the start: simple, exact odds, easy to audit
Softmax (Boltzmann)	pick each move with probability proportional to exp(score / temperature)	exact, from the scores	tuned by temperature; bad moves rarely picked	once scores are calibrated
Thompson sampling	sample each move's reply rate from its posterior and take the highest	computed by simulation; log it at decision time	low; explores where it's unsure	when there are many plays and a learning loop
Upper confidence bound (LinUCB)	take the move with the highest optimistic estimate	deterministic, so 0 or 1: bad for OPE	low	avoid unless a randomized slice is added on top
Randomized A/B slice	a fixed share of leads gets a different policy entirely	exact per arm	the share's lost replies	to confirm a policy that already won offline

the objects

A move log row, and a policy evaluation.

The move row is written when the move is chosen, before anything is sent. The reward fields are filled when the window closes. The policy evaluation is an experiment record for an offline test: which logs, which estimator, which policy, and the rule for going live.

Field	Type	What it holds
`move_id`	string	Becomes the touch_id when the message is sent, so the reward joins without guessing.
`lead_id`	string	Who the move is for.
`context`	JSON, frozen	Exactly what the picker saw: lead type, triggers and ages, fit lines, tier, past touches, channel availability.
`allowed`	list	Plays the contact policy allowed for this lead right now.
`candidates`	list	Every allowed play with its score from the policy.
`chosen`	play id	The move taken.
`propensity`	0 to 1	The probability the policy gave the chosen move. Required.
`policy_version`	version string	Which picker made the choice.
`explore`	boolean	Whether this move came from the exploration slice.
`decided_at`	timestamp	When the move was chosen.
`reward`	object	Filled when the window closes: replied_7d, signed_up_14d, stop_request.
`reward_closed_at`	timestamp	When the window closed. Rows without it are excluded.

one move, as data

{
 "move_id": "mv_01J…",
 "lead_id": "lead_example",
 "context": {
 "lead_type": "cold pain post",
 "trigger": "asks for a recommendation",
 "signal_age_h": 6,
 "tier": "Hot",
 "fit_lines_matched": [
 "live SaaS",
 "solo founder"
 ],
 "past_touches": 0,
 "channels": [
 "x",
 "reddit"
 ]
 },
 "allowed": [
 "public reply",
 "DM"
 ],
 "candidates": [
 {
 "play": "public reply",
 "score": 0.62
 },
 {
 "play": "DM",
 "score": 0.31
 }
 ],
 "chosen": "public reply",
 "propensity": 0.95,
 "policy_version": "nba_v1",
 "explore": false,
 "decided_at": "2026-10-20T13:04:00Z",
 "reward": null,
 "reward_closed_at": null
}

the policy evaluation

Field	Type	What it holds
`eval_id`	string	`ope.<policy>.<n>`
`target_policy`	version string	The candidate being judged.
`logging_policies`	version strings	Which pickers produced the logs.
`log_window`	dates	Moves included; only rows with closed reward windows.
`estimators`	list	SNIPS and DR by default, with clipping if used.
`estimates`	per estimator	Value and 90% interval (bootstrap over moves).
`ess`	number	Effective sample size. Below a pre-set floor, the evaluation can't pass.
`baseline`	number	The logging policy's own observed reward.
`rule`	text	The go-live rule, set before computing.
`decision`	go live on a slice \| keep \| collect more	What happened next.

one evaluation, as data

{
 "eval_id": "ope.recommender_v1.01",
 "target_policy": "recommender_v1",
 "logging_policies": [
 "nba_v1 + 10% uniform slice"
 ],
 "log_window": "YYYY-MM-DD/YYYY-MM-DD",
 "estimators": [
 "SNIPS",
 "DR"
 ],
 "estimates": null,
 "ess": null,
 "baseline": null,
 "rule": "go live on a randomized half of leads if DR beats the baseline and ESS > 200",
 "decision": null
}

catalog

Estimators, policies and safeguards.

estimator

Inverse propensity scoring (IPS)

what: Unbiased if propensities are logged correctly. Variance grows with large weights.
use: rows where the new policy agrees with the logged move, re-weighted

estimator

Self-normalized IPS (SNIPS)

what: Divides by the sum of weights instead of n. Slight bias, much lower variance on small logs.
use: the default first estimate

estimator

Direct method

what: Fits a reward model and asks it about every action. Biased whenever the model is wrong.
use: a sanity check next to IPS

estimator

Doubly robust

what: Direct method plus an IPS correction on the model's errors. Right if either part is right.
use: the default once a reward model exists

estimator

Clipped IPS

what: Caps each weight (for example at 20). Trades a little bias for much less variance.
use: logs with a few rare moves

safeguard

Effective sample size

what: (Σw)² / Σw² says how many rows really carry the estimate.
use: a floor in the go-live rule

safeguard

Bootstrap interval

what: Resample moves 1,000 times and recompute; report the middle 90%.
use: every estimate, always

safeguard

Allowed-set logging

what: Only moves the contact policy allowed are actions. Blocked ones are neither explored nor scored.
use: safety and correct support

policy

ε-greedy slice

what: Top move most of the time, uniform among allowed moves ε of the time.
use: the first exploration policy

policy

Thompson sampling

what: Picks in proportion to how likely each move is to be best.
use: many plays, a learning loop

policy

Contextual bandit recommender

what: A model of reward by context and play; picks the best expected move with some exploration.
use: what 'recommender v1' becomes

safeguard

Fixed reward windows

what: Replied within 7 days, signed up within 14; open windows are excluded.
use: fair comparison across weeks

failure modes

How OPE goes wrong, and how to catch it.

No propensity logged — Moves were chosen by a ranker, but the probability wasn't written. · Make propensity a required column; a move row without it fails to insert.

Deterministic picker — Every move had probability 1, so nothing else is evaluable. · Keep a permanent exploration slice among allowed moves.

Huge weights — A few rare moves dominate the estimate. · Report the effective sample size; clip weights; prefer SNIPS or DR.

Context recomputed — The new policy is scored on features it wouldn't have had. · Store context as frozen JSON on the move row.

Open reward windows — Recent moves haven't had 7 days to get a reply, so they drag the estimate down. · Only rows with reward_closed_at set are scored.

Exploring unsafe moves — Random picks include a DM to someone who said stop. · Explore only within the allowed set; never-contact leads get no moves at all.

Non-stationary world — Last month's reply rates don't hold after a platform or source change. · Weight recent logs; confirm offline wins on a small live slice.

Reward leakage — The reward includes replies to a later move, not this one. · Join rewards by move id (the touch_id), not by lead and date.

data to capture

What the move log records, and what that trains.

Record — every move with frozen context, allowed set, candidates and scores, the chosen move, its propensity, policy version and exploration flag; rewards joined on the move id when the window closes; every policy evaluation with estimator, ESS and rule

Label — replied within 7 days, signed up within 14, stop request; each joined to exactly one move

Trains — the playbook recommender (a contextual bandit), a reward model for the direct and doubly robust estimators, and calibrated move scores

Test — new pickers are scored offline first; only winners get a randomized live slice with their own logged odds

Counterfactual evaluation from logged bandit feedback.

Log the odds at decision time, re-weight later.

Four ways to turn logs into an estimate.

Judge a new playbook picker on logs from the old one.

You can only evaluate moves the old policy sometimes made.

Pay a little now to be able to learn later.

A move log row, and a policy evaluation.

Estimators, policies and safeguards.

Inverse propensity scoring (IPS)

Self-normalized IPS (SNIPS)

Direct method

Doubly robust

Clipped IPS

Effective sample size

Bootstrap interval

Allowed-set logging

ε-greedy slice

Thompson sampling

Contextual bandit recommender

Fixed reward windows

How OPE goes wrong, and how to catch it.

What the move log records, and what that trains.

Where moves come from and where rewards land.