all guides

Off-policy evaluation

Judging a new policy on logged decisions.

learn · experimentation engine

The next-best-action engine picks one move per lead: reply in public, DM, email, wait. A smarter picker could reply more often, but testing it live means trying worse moves on real people. Off-policy evaluation estimates how a new picker would have done, using only the old picker's logged decisions, as long as each decision was logged with the probability it had of being chosen. That probability is the whole trick, and most systems never record it.

context → logged move + its odds → reward → estimate a new policy offline → small live slice → ship

Counterfactual evaluation from logged bandit feedback.

what it is, in industry terms

Recommendation, ads and search teams use off-policy evaluation (OPE) to judge a new ranking or bidding policy before it serves anyone. The setting is called a contextual bandit. For each unit you see a context, take one action, and observe a reward for that action only. You never see what the other actions would have earned. OPE re-weights the logged rewards so they speak for a different policy.

Context — x · What the picker knew: lead type, trigger, freshness, fit lines, channel, past touches.

Action — a · The move it chose: a play, a channel, an opener, or wait.

Reward — r · The outcome in a fixed window: replied within 7 days, signed up within 14.

Logging policy — π₀ · behavior policy · The picker that actually ran, with the probability it gave each action.

Target policy — π · candidate · The new picker you want to judge without running it.

Propensity — π₀(a | x) · The probability the logging policy gave the action it took. Logged at decision time.

Importance weight — w = π(a|x) / π₀(a|x) · How much more or less the new policy would have taken this action.

Support — overlap · Every action the new policy takes must have had a nonzero chance under the old one.

Log the odds at decision time, re-weight later.

how it works

Each move is logged with the context the picker saw, every candidate it considered with its score, the action chosen and its probability. When the reward window closes, the reward is joined on the move id. A candidate policy is then scored against the same rows: rows where it agrees with what happened count for more or less, depending on how likely that action was. Rows where it would have done something else count for nothing.

  1. Context x — lead type, trigger, freshness, fit, history
  2. Allowed — set contact policy removes blocked moves
  3. Old policy — π₀ picks a with probability π₀(a∣x), logged
  4. Log the — move row before anything is sent
  5. Reward r — replied within 7 days, joined on move id
  6. Score new — policy π weight w = π / π₀ IPS · SNIPS · DR
  7. Live slice — if it wins randomized, with its own logged odds

Four ways to turn logs into an estimate.

the estimators

IPS V̂ = (1/n) Σᵢ wᵢ · rᵢ wᵢ = π(aᵢ|xᵢ) / π₀(aᵢ|xᵢ) SNIPS V̂ = Σᵢ wᵢ · rᵢ / Σᵢ wᵢ self-normalized: lower variance, slight bias DM V̂ = (1/n) Σᵢ Σₐ π(a|xᵢ) · r̂(xᵢ, a) a reward model r̂ fills in every action DR V̂ = DM + (1/n) Σᵢ wᵢ · (rᵢ − r̂(xᵢ, aᵢ)) doubly robust: right if either π₀ or r̂ is right ESS = (Σ wᵢ)² / Σ wᵢ² effective sample size: how many rows really count

EstimatorBiasVarianceNeedsUse it when
IPSnone, if propensities are righthigh, explodes with small propensitieslogged propensitiesmany rows and moderate weights
SNIPSsmall, shrinks with more rowslower than IPSlogged propensitiesthe default for small logs
Direct method (DM)as large as the reward model's errorlowa reward model r̂(x, a)as a sanity check, or with no exploration (and then distrust it)
Doubly robust (DR)none if either the propensities or r̂ is rightusually lowestboththe default once a reward model exists

Judge a new playbook picker on logs from the old one.

worked example

A toy version of a playbook picker's choice: three kinds of lead, three plays. The old picker DMs everyone, except for an exploration slice where it picks uniformly at random. The new picker sends each lead type its best play. Draw logs from the old picker, estimates the new picker's reply rate four ways, and repeats 200 times to show how much each estimate wobbles. Reply rates are invented for illustration.

logging setup

An exploration slice ε of 10% (a uniform random play) and 1,000 logged moves.

Lead type · sharepublic replyDMemail
cold pain post · 50%12%5%2%
warm signup, stalled · 30%4%10%18%
competitor-unhappy · 20%8%15%6%

True reply rates per play (illustrative). Bold is the new picker's choice. The old picker DMs every type.

estimated reply rate of the new picker · 200 repeats

14.4% — true reply rate of the new picker

8.5% — reply rate the old picker got

41 — effective sample size of 1,000 logs

IPS — 14.7%

SNIPS — 14.7%

DM — 14.7%

DR — 14.7%

Each figure is the average estimate across the 200 repeats; all four land close to the true 14.4%, and the effective sample size of 41 says how few of the 1,000 logs actually inform the new picker.

with ε = 0

With no exploration, the old picker never sends a public reply or an email, so the logs say nothing about the new picker's main moves. IPS and SNIPS collapse, and the direct method can only guess. That's the support problem, and no estimator fixes it after the fact.

You can only evaluate moves the old policy sometimes made.

support and overlap

  • Deterministic logging kills OPE. A picker that always takes its top move logs a propensity of 1 for it and 0 for everything else. Any new policy that differs anywhere has no data to stand on.
  • Small propensities explode variance. A move taken with 2% probability gets a weight of 50. A handful of such rows decide the estimate. Watch the effective sample size, and clip weights (cap w at, say, 20) with the bias that brings.
  • Propensities must be logged, not reconstructed. Recomputing "what the probability would have been" later, from a changed scorer, is guessing. Write it at decision time with the policy version.
  • Context must be frozen. If the context is recomputed later, the new policy is scored on information it wouldn't have had (rule 3 again).
  • Rewards need a fixed window. "Replied within 7 days" must be closed before a row counts, or recent moves look worse than old ones.
  • Blocked moves aren't actions. If the contact policy forbade a move for this lead, neither policy could take it. Log the allowed set, and explore only inside it.

Pay a little now to be able to learn later.

exploration

Exploration means sometimes taking a move the picker doesn't rank first, on purpose and at a logged probability. It costs some replies today. It buys the ability to evaluate any future picker offline, and to learn moves nobody has tried. The safety rule: explore only among moves the contact policy allows for that lead, and never on leads marked never-contact.

StrategyHow it picksPropensityCostFit for a small founder
Uniform slice (ε-greedy)with probability ε pick any allowed move at random; otherwise the top moveexact: 1 − ε + ε/k for the top move, ε/k for each otherabout ε of moves are not the best guessthe start: simple, exact odds, easy to audit
Softmax (Boltzmann)pick each move with probability proportional to exp(score / temperature)exact, from the scorestuned by temperature; bad moves rarely pickedonce scores are calibrated
Thompson samplingsample each move's reply rate from its posterior and take the highestcomputed by simulation; log it at decision timelow; explores where it's unsurewhen there are many plays and a learning loop
Upper confidence bound (LinUCB)take the move with the highest optimistic estimatedeterministic, so 0 or 1: bad for OPElowavoid unless a randomized slice is added on top
Randomized A/B slicea fixed share of leads gets a different policy entirelyexact per armthe share's lost repliesto confirm a policy that already won offline

A move log row, and a policy evaluation.

the objects

The move row is written when the move is chosen, before anything is sent. The reward fields are filled when the window closes. The policy evaluation is an experiment record for an offline test: which logs, which estimator, which policy, and the rule for going live.

FieldTypeWhat it holds
move_idstringBecomes the touch_id when the message is sent, so the reward joins without guessing.
lead_idstringWho the move is for.
contextJSON, frozenExactly what the picker saw: lead type, triggers and ages, fit lines, tier, past touches, channel availability.
allowedlistPlays the contact policy allowed for this lead right now.
candidateslistEvery allowed play with its score from the policy.
chosenplay idThe move taken.
propensity0 to 1The probability the policy gave the chosen move. Required.
policy_versionversion stringWhich picker made the choice.
explorebooleanWhether this move came from the exploration slice.
decided_attimestampWhen the move was chosen.
rewardobjectFilled when the window closes: replied_7d, signed_up_14d, stop_request.
reward_closed_attimestampWhen the window closed. Rows without it are excluded.

one move, as data

{
 "move_id": "mv_01J…",
 "lead_id": "lead_example",
 "context": {
 "lead_type": "cold pain post",
 "trigger": "asks for a recommendation",
 "signal_age_h": 6,
 "tier": "Hot",
 "fit_lines_matched": [
 "live SaaS",
 "solo founder"
 ],
 "past_touches": 0,
 "channels": [
 "x",
 "reddit"
 ]
 },
 "allowed": [
 "public reply",
 "DM"
 ],
 "candidates": [
 {
 "play": "public reply",
 "score": 0.62
 },
 {
 "play": "DM",
 "score": 0.31
 }
 ],
 "chosen": "public reply",
 "propensity": 0.95,
 "policy_version": "nba_v1",
 "explore": false,
 "decided_at": "2026-10-20T13:04:00Z",
 "reward": null,
 "reward_closed_at": null
}

the policy evaluation

FieldTypeWhat it holds
eval_idstringope.<policy>.<n>
target_policyversion stringThe candidate being judged.
logging_policiesversion stringsWhich pickers produced the logs.
log_windowdatesMoves included; only rows with closed reward windows.
estimatorslistSNIPS and DR by default, with clipping if used.
estimatesper estimatorValue and 90% interval (bootstrap over moves).
essnumberEffective sample size. Below a pre-set floor, the evaluation can't pass.
baselinenumberThe logging policy's own observed reward.
ruletextThe go-live rule, set before computing.
decisiongo live on a slice | keep | collect moreWhat happened next.

one evaluation, as data

{
 "eval_id": "ope.recommender_v1.01",
 "target_policy": "recommender_v1",
 "logging_policies": [
 "nba_v1 + 10% uniform slice"
 ],
 "log_window": "YYYY-MM-DD/YYYY-MM-DD",
 "estimators": [
 "SNIPS",
 "DR"
 ],
 "estimates": null,
 "ess": null,
 "baseline": null,
 "rule": "go live on a randomized half of leads if DR beats the baseline and ESS > 200",
 "decision": null
}

Estimators, policies and safeguards.

catalog

estimator

Inverse propensity scoring (IPS)

what
Unbiased if propensities are logged correctly. Variance grows with large weights.
use
rows where the new policy agrees with the logged move, re-weighted

estimator

Self-normalized IPS (SNIPS)

what
Divides by the sum of weights instead of n. Slight bias, much lower variance on small logs.
use
the default first estimate

estimator

Direct method

what
Fits a reward model and asks it about every action. Biased whenever the model is wrong.
use
a sanity check next to IPS

estimator

Doubly robust

what
Direct method plus an IPS correction on the model's errors. Right if either part is right.
use
the default once a reward model exists

estimator

Clipped IPS

what
Caps each weight (for example at 20). Trades a little bias for much less variance.
use
logs with a few rare moves

safeguard

Effective sample size

what
(Σw)² / Σw² says how many rows really carry the estimate.
use
a floor in the go-live rule

safeguard

Bootstrap interval

what
Resample moves 1,000 times and recompute; report the middle 90%.
use
every estimate, always

safeguard

Allowed-set logging

what
Only moves the contact policy allowed are actions. Blocked ones are neither explored nor scored.
use
safety and correct support

policy

ε-greedy slice

what
Top move most of the time, uniform among allowed moves ε of the time.
use
the first exploration policy

policy

Thompson sampling

what
Picks in proportion to how likely each move is to be best.
use
many plays, a learning loop

policy

Contextual bandit recommender

what
A model of reward by context and play; picks the best expected move with some exploration.
use
what 'recommender v1' becomes

safeguard

Fixed reward windows

what
Replied within 7 days, signed up within 14; open windows are excluded.
use
fair comparison across weeks

How OPE goes wrong, and how to catch it.

failure modes

No propensity logged — Moves were chosen by a ranker, but the probability wasn't written. · Make propensity a required column; a move row without it fails to insert.

Deterministic picker — Every move had probability 1, so nothing else is evaluable. · Keep a permanent exploration slice among allowed moves.

Huge weights — A few rare moves dominate the estimate. · Report the effective sample size; clip weights; prefer SNIPS or DR.

Context recomputed — The new policy is scored on features it wouldn't have had. · Store context as frozen JSON on the move row.

Open reward windows — Recent moves haven't had 7 days to get a reply, so they drag the estimate down. · Only rows with reward_closed_at set are scored.

Exploring unsafe moves — Random picks include a DM to someone who said stop. · Explore only within the allowed set; never-contact leads get no moves at all.

Non-stationary world — Last month's reply rates don't hold after a platform or source change. · Weight recent logs; confirm offline wins on a small live slice.

Reward leakage — The reward includes replies to a later move, not this one. · Join rewards by move id (the touch_id), not by lead and date.

What the move log records, and what that trains.

data to capture

Record — every move with frozen context, allowed set, candidates and scores, the chosen move, its propensity, policy version and exploration flag; rewards joined on the move id when the window closes; every policy evaluation with estimator, ESS and rule

Label — replied within 7 days, signed up within 14, stop request; each joined to exactly one move

Trains — the playbook recommender (a contextual bandit), a reward model for the direct and doubly robust estimators, and calibrated move scores

Test — new pickers are scored offline first; only winners get a randomized live slice with their own logged odds

More in Experimentation engine