Autotune

How to monitor your Autotune test

Computations and traffic allocation

The results tab within Autotune provides a view of your ongoing and completed Autotune tests. Statsig computes Autotune hourly, with metrics and traffic allocation updated throughout the day. Statsig ingests results from exposures after the attribution window is complete. For example, Statsig doesn't count an exposure at 1pm with a 6-hour attribution window until 7pm.

Result

This section shows a summary of your Autotune test. The top bars show a 95% Bayesian Credible interval for the estimated conversion rate (exposure to success event). There's a 95% chance that the real value is within this interval. The table shows the number of exposures, success events, and overall success rate for each variation across the duration of the test. It also includes a plain-language description of the current test state.

Details

There are several charts provided:

1. Probability of Best - shows the progress of the Autotune test, and which variant is currently winning.
2. Cumulative success rate - shows the overall success rate (exposure -> success) to date.
3. Daily success rate - shows the success rate for each variation per day.
4. Traffic allocation - shows the daily number of exposures allocated to each variation on a given day.

Linked experiments

The most reliable way to evaluate whether a bandit is working is to measure whether it drives more of the targeted behavior compared to a baseline experience. You can set up and link an A/B test in Statsig to evaluate this, which also lets you monitor other user behaviors and guardrail metrics.

Linking an A/B test is the most rigorous form of measurement, and Statsig highly encourages it.

Standard practice is to wrap the Autotune in an experiment with a binary parameter. The wrapping experiment has two groups: a treatment group that receives the Autotune, and a control group that receives a default experience. Only users in the treatment group reach the Autotune, which then allocates them across its own variants. This is why the wrapping experiment and the Autotune can have a different number of variants.

Common splits are 50/50 or a 90/10 holdback, where 90% of users receive the Autotune and 10% receive the default experience. For example, a 90/10 holdback around an Autotune with three variants splits traffic like this:

90% → Autotune
      ├─ Variant A
      ├─ Variant B
      └─ Variant C
10% → Default experience

The two results answer different questions:

• The wrapping experiment measures whether the Autotune outperforms the default experience. Its treatment result combines all Autotune variants.
• The Autotune results show which variant performs best and how Statsig allocates traffic among variants.

Linking the experiment to the Autotune only displays the experiment's results on the Autotune page. Linking doesn't route traffic. Your code must check the experiment and call the Autotune only for users in the treatment group:

experiment_value = statsig.get_experiment('wrapping_experiment').get('flag')
default_param = '...'
if experiment_value:
  param = statsig.get_experiment('autotune').get('param_name')
else:
  param = default_param

# use param in code

Start this experiment at the same time you launch your Autotune.