Autotune Methodology
How multi-armed bandits work in Statsig Autotune to automatically allocate traffic to the best- performing variant based on a single goal metric.
The base Autotune implementation uses a Thompson Sampling (Bayesian) algorithm to estimate each variant's probability of being the best variant and allocate a proportional amount of traffic.
For example, if a given variant has a 60% probability of being the best, Autotune allocates 60% of the traffic to it. Autotune doesn't wait for a variant to be clearly better before shifting traffic. Statsig recomputes these probabilities hourly, so traffic moves gradually toward the variants that perform best on the target metric as evidence accumulates.
Allocation starts with the exploration window, when Autotune splits traffic equally across variants before it has enough data. After that, higher-performing variants receive more traffic and underperforming variants receive less. The process ends when one variant's probability of being the best reaches the Autotune's Winner Threshold (80%, 90%, 95%, 98%, or 99%). Autotune then declares that variant the winner, stops collecting data, and directs all traffic to it. You set the exploration window and Winner Threshold when you set up the Autotune, and you can watch each variant's probability of being the best in the Probability of Best chart.
How probability of best is calculated
When Autotune optimizes for whether a success event happens, it models each variant's true success rate as a Beta distribution built from that variant's cumulative successes and failures, Beta(successes + 1, failures + 1). The distribution starts from a flat prior and narrows as attempts build up. The 95% credible interval on the Results tab covers the 2.5th to the 97.5th percentile of this distribution.
A variant's probability of best is the probability that its true success rate is higher than every other variant's at the same time. Statsig calculates it numerically from all variants' distributions together, so the values across variants add up to 100%. Autotune compares this probability directly with the Winner Threshold, with no p-value or significance test involved.
Keep the following in mind when you read the results:
- Probability of best compares all variants at once, so a variant can pass the Winner Threshold while the variants' credible intervals still overlap.
- Autotune doesn't adjust for the number of variants, and it checks the Winner Threshold every hour. A variant only needs to pass the threshold once to be declared the winner, so more variants and lower thresholds raise the chance of declaring a variant that isn't the true best.
- The model treats each attempt as independent and doesn't adjust for users who return or see more than one variant. When many users return, the credible intervals can be narrower than they would be in a per-user analysis.
Some helpful references:
- Introducing Autotune: Statsig's Multi-Armed Bandit (Statsig blog)
- Analysis of Thompson Sampling for the Multi-armed Bandit Problem (Agrawal and Goyal, COLT 2012), a regret analysis of Thompson Sampling
Advantages
The main advantage of the base Multi-Armed Bandit over a contextual bandit is its ability to converge and identify the best variant. When a single solution works well for all users, the Multi-Armed Bandit efficiently allocates traffic and determines the correct long-term solution while minimizing regret. Regret is the cost of exposing users to a worse variant. A fixed-split A/B test keeps sending a share of users to losing variants until the test ends.
Disadvantages
The main disadvantage of a Multi-Armed Bandit compared to a contextual bandit is its inability to personalize. When user attributes interact with variants, Autotune can identify a global maximum that is worse than serving each user their individual best variant.
For example, suppose you test flag images on a landing page for users in the US and Canada. Even if the "US Flag" variant has the highest overall click-through rate (CTR), it's a poor choice for Canadian users. Autotune still serves the US Flag to everyone, so Canadian users get a worse experience than a contextual bandit would give them.
| A/B/n Test | Multi-Armed Bandit (Autotune) | Contextual Bandit (Autotune AI) | Ranking Engine | |
|---|---|---|---|---|
| Typical # Variants | 2-3 | 4-8 | 4-8 | Arbitrary # |
| Personalization Factor | None | None | Moderate | High |
| Input Data Required | None | Very Little (100+ samples) | Little - generally 1000+ samples | Tens of thousands to millions of samples |
| Model Efficacy | None | Basic | Moderate | High |
| Identifies Best Variant | Yes | Yes | No | No |
| Consistent User Assignment | Yes | No | No | No |
Was this helpful?
For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.