Whom should a retailer e-mail? On this experiment, everyone.

At the default economics, no targeting rule beats e-mailing everyone

ADid the experiment work, and what did it do on average?

The randomisation holds: arm sizes match the 1/3 design (chi-square p = 0.90) and every covariate is balanced (largest absolute standardised difference 0.014, against a usual alarm level of 0.1). Both e-mails work on average.

Average effects, three estimators

Variance reduction barely helps here. CUPED on past-year spend narrows the spend interval by 0.02%; the fully interacted regression adjustment of Lin (2013) narrows the visit interval by 1.5%. A year of purchase history explains almost nothing of a two-week window, so the plain difference in means is the honest headline.

Pre-declared segments

Five segment dimensions were declared before any model was fitted and tested once per e-mail and outcome: 30 heterogeneity tests, Holm-corrected. 2 of 30 survive, all on visits: the men's e-mail lifts the visit rate by +13.4 pp among buyers of both lines, against +6.9 pp to +7.1 pp for the other product-line groups; the women's e-mail lifts the visit rate by +1.1 pp among menswear-only buyers, against +7.1 pp to +7.4 pp for the other product-line groups. Purchases and spend show no detectable heterogeneity at this sample size.

BCan a model find the persuadable customers?

Five meta-learners (S, T, X, cross-fitted DR, transformed outcome), each with a LightGBM and a regularised linear base model, trained with five-fold cross-fitting so that every customer is scored by models that never saw them, and benchmarked against the response model a marketing team would build.

Where the effect is decoupled from the baseline, as with the women's e-mail, the uplift learners find it: the DR-learner's Qini coefficient on visits is 2.7 [1.7, 3.6] (×1,000, bootstrap 95% CI), clearly above random. The plain response model finds much of it too (2.0 [1.1, 2.9]): paired on the same bootstrap draws, the DR-learner does not detectably beat it (+0.7 [-0.2, +1.7]), although 9 of 10 uplift learners score above it on point estimate. Where the effect grows with the baseline, as with the men's e-mail, the DR-learner is no better than random (0.1 [-0.8, 1.1]), loses to the response model (-1.6 [-2.7, -0.6]), and only 1 of 10 uplift learners score above the response model. On spend, 0 of 22 rankings have a Qini interval that excludes zero, and all 11 point estimates for the men's e-mail are negative.

Where the response model wastes contacts

For the women's e-mail, the response model ranks customers by how likely they are to visit, so its top deciles are full of people who would have come anyway. In its top 20%, 71% of the e-mailed visitors would have visited without the e-mail, against 58% in the top 20% of the uplift ranking (T-learner, LightGBM), which buys 8.4 pp of extra visit rate there against 6.8 pp. Reading the control rate as "would have come anyway" assumes the e-mail deters no one.

CWhich policy should ship?

The decision is scored on spend, because profit depends on spend. Every policy value is a doubly robust off-policy estimate on customers that neither trained the models nor chose the policy, with a paired standard error for every comparison.

30 candidate rankings compete: five learners and two base models, trained on visits, conversions or spend. A model trained on dense visits might rank customers better for revenue than one trained on sparse revenue, so the choice is left to data, but to data the final score never sees. The candidates' merit barely replicates from one random half of the customers to the other: rank correlation 0.24 on spend, against 0.66 on visits. With 578 buyers, the best-looking model is mostly the luckiest one.

The table values come from one random split of the customers, and their intervals are conditional on the rankings and budgets that split selected. Re-running the whole procedure on 50 random splits, the uplift procedure averages -$41 per 1,000 against blanket sending (from -$89 to +$9), falls short of it in 48 of 50 splits, e-mails a median 80% of customers (52% to 100%), and settles on 16 different rankings along the way.

DWhat changes when the experiment is two hundred times larger?

The Criteo Uplift v2.1 benchmark logs 13,979,592 users from randomised advertising incrementality tests, 85% treated. The same learners were fitted on a hash-based sample of 2.3M users from one half and evaluated on all 6,994,996 users of the other half. Treatment lifts visits by +1.03 pp and conversions by +0.115 pp on average.

At this size every ranking beats random by a wide margin, and the effect is concentrated: the top 10% of users by the response model hold 51% of all incremental visits. Their untreated visit rate is 32% and treatment adds +5.3 pp: here the effect tracks the baseline rate, so ranking by "who will act" is also ranking by "who will be moved". The response model is the best uplift ranker (Qini ×1,000 on conversions: 0.43, against 0.28 for the T-learner and 0.28 for the DR-learner). The meta-learners pay for estimating a small difference between two large, noisy predictions; with 15% controls and a control conversion rate of 0.19%, the DR-learner's final stage returns the same score for 63% of users. The 0.7% of users it scores below zero convert at 5.6% without treatment, against 0.19% overall, and gain +2.0 pp when treated: high-baseline users whose noisy pseudo-outcomes pushed their scores below zero. They hold 13% of all incremental conversions, which is why its curve jumps in the last 1%.

Method and limits

• Out-of-sample by construction. Five outer folds; every score comes from models fitted without that customer. The DR-learner cross-fits its own nuisance models inside the training folds. A test scrambles held-out outcomes and checks that held-out scores do not move.
• Known propensities. Assignment probabilities are fixed by design (1/3 per arm), so IPW and doubly robust policy values are unbiased without a propensity model. Tests check both on synthetic data, the DR one with a deliberately wrong outcome model.
• Selection is part of the model. Ranking and budget are chosen on one random half and applied to the other; the two halves are pooled.
• Uncertainty. Qini intervals are a Poisson bootstrap over customers with scores held fixed: they cover evaluation noise, not retraining noise. Policy intervals are normal approximations on per-customer doubly robust scores, conditional on the rankings and budgets one split selected; the 50-split repeat shows the selection noise they leave out.
• Assumed economics. The data has no margins or costs. The defaults (40% gross margin, $0.15 per e-mail including fatigue and unsubscribe risk) are assumptions; the sliders show the conclusions under others.
• Limits. One two-week campaign from 2008 with 578 buyers; spend is heavy-tailed. Criteo features are anonymised, its labels are visits and conversions (not revenue), and its Qini intervals use 20 random groups of the held-out half instead of a bootstrap. A multiplicative model (response × relative lift) was not tried.