When everything moves together, what should you remember?
The same structure, in many fields
Run a comparative experiment long enough and something will move everything you are comparing, regardless of the difference you care about.
- Online A/B tests: promotions, weekdays and app releases lift or drop every variation’s conversion rate together.
- Advertising: time of day and season shake every creative’s click rate.
- Email and push: the day and hour of sending move every message’s response rate.
- Mobile health: a person’s day, mood and schedule move how likely they are to respond to any prompt.
- Long clinical trials: calendar time, site and changes in standard of care shift outcomes in every arm.
- Agricultural field trials: soil differs from plot to plot, so every variety in a good block does well together.
The last one is the clue. One of the oldest answers in statistics is blocking: compare only within a block, and take the block’s own effect out of the comparison. What moves everything is the level; what you want to know is the difference on top of it.
The scale of that difference matters too. When the level moves, a difference in probabilities and an odds ratio cannot both stay fixed. In meta-analyses of clinical trials, relative measures such as the odds ratio have long been reported to be more consistent than risk differences across studies with different baseline risk (Deeks, 2002). The method below makes its bet on that relative scale: on the log-odds scale, the differences hold while the level moves.
Adaptive allocation makes it worse
With a fixed 50/50 split, a moving level does little harm. Every variation goes through the same periods in the same proportion, so the movement cancels.
A bandit is different. Moving traffic toward whatever looks better is its whole job, so each variation ends up living through different periods. Rates pooled over the run tilt toward whichever variation got more traffic in the good stretch. Luck, not quality, reorders them.
Carrying blocking over to bandits gives the answer. Treat each batch as a block. Fit a logistic regression to the batch, let a separate intercept absorb that batch’s level and throw it away, and carry only the log-odds differences between variations to the next batch. I wrote this up as Odds-Ratio Thompson Sampling (OR-TS) and released it as pip install orts.
Measured: where it holds and where it doesn’t
Whether the bet holds can be measured: estimate the level and the differences in every period, and see how much each moves beyond sampling noise. Few public datasets allow this. I used two.
- It held: 71 A/B tests from one online retailer (the public ASOS dataset). Between periods the level moved by about a third of a logit, while the differences between variations were around a hundredth of that in size. In all 86 series the level moved more than the difference. Whether the differences persist over long spans is not settled by this data.
- It failed: one fashion recommendation log (the Open Bandit dataset). Here the differences moved more than the level, because a few items rose and fell in popularity within the week. In that environment no method was clearly better than another.
So this is not a law that holds everywhere. It is a property to check in each field, and checking takes minutes.
How to check your own data
All you need is exposures and events per period and variation, which is what an experimentation platform’s aggregate table already holds.
from orts import batches_from_rows
from orts.diagnostics import batch_contrasts, excess_sd
levels, contrasts = [], []
for obs in batches_from_rows(rows): # one row per (period, variation)
level, effects = batch_contrasts(obs, reference="control")
levels.append(level)
contrasts.append(effects["treatment"])
print("level moves by ", excess_sd(*zip(*levels)))
print("difference moves by", excess_sd(*zip(*contrasts)))Both numbers are the standard deviation across periods with sampling noise removed, in logits. If the first is much larger than the second, throwing away the level and remembering only the differences is the right bet in your setting. If the second is large, the differences move over time as well, and you need a way to forget the past gradually (decay).
Against real product code
To see how much it matters where the bet holds, I compared against the bandit engine of GrowthBook, an open-source experimentation platform. Because the engine is public, I could run the actual product code. It uses another form of blocking: take each period’s mean and combine the periods with the same weights for every variation.
Binary outcomes, 40 periods of 20,000 users, five variations, 20 runs. Cumulative regret counts the conversions lost against always serving the best variation; lower is better.
| Common shifts | No shifts | |
|---|---|---|
| Period weighting (GrowthBook) | 642 | 622 |
| Pooled over the run, no weighting | 798 | 400 |
| Level dropped every period (OR-TS) | 421 | 360 |
Period weighting protects against shifts (642 against 798) but does worse than pooling when nothing moves (622 against 400), because its weights follow each period’s share of all traffic rather than how many users that variation got. Dropping the level weights each observation by the information it carries, so it gets the protection without that price.
I also extended it to user groups whose best variation differs: drop the level for every group and period, keep a difference per group, and let thin groups borrow from the others. I committed four hypotheses and their failure criteria before running anything. Three held and one failed. The failure was using the leaves of a tree that splits on outcome levels as the groups: once groups with different best variations share a leaf, nothing can separate them again. It is the original lesson of blocking: form the blocks around the comparison, not around the outcome.
What this is and isn’t
The comparisons are simulations, with binary outcomes only. The real-data measurements cover two datasets; the bet held in one and failed in the other. Advertising, mobile health, clinical trials and field trials share the structure; I have not measured them. If someone with data from those fields runs the check above, that will be the best answer to how far this goes.
Nor does any of this say period weighting is wrong. It works for mean metrics too and is simple to compute. Dropping the level is, for now, only available for binary outcomes.
Thanks to the GrowthBook team for keeping their engine open and for reviewing my proposal carefully.