Autotune

How to monitor your Autotune test

Computations and traffic allocation

The results tab within Autotune provides a view of your ongoing and completed Autotune tests. Statsig computes Autotune hourly, with metrics and traffic allocation updated throughout the day. Statsig ingests results from exposures after the attribution window is complete. For example, Statsig doesn't count an exposure at 1pm with a 6-hour attribution window until 7pm.

Result

This section shows a summary of your Autotune test. The top bars show a 95% Bayesian Credible interval for the estimated conversion rate (exposure to success event). There's a 95% chance that the real value is within this interval. The table shows the number of exposures, success events, and overall success rate for each variation across the duration of the test. It also includes a plain-language description of the current test state.

Details

There are several charts provided:

1. Probability of Best - shows the progress of the Autotune test, and which variant is currently winning.
2. Cumulative success rate - shows the overall success rate (exposure -> success) to date.
3. Daily success rate - shows the success rate for each variation per day.
4. Traffic allocation - shows the daily number of exposures allocated to each variation on a given day.

Linked experiments

The most reliable way to evaluate whether a bandit is working is to measure whether it drives more of the targeted behavior compared to a baseline experience. You can set up and link an A/B test in Statsig to evaluate this, which also lets you monitor other user behaviors and guardrail metrics.

Linking an A/B test is the most rigorous form of measurement, and Statsig highly encourages it.

Standard practice is to wrap the autotune in an experiment with a binary parameter, either as 50/50 or a 90/10 holdback. You can link the experiment to the autotune to get the results on the autotune page. In code, this might look like:

experiment_value = statsig.get_experiment('wrapping_experiment').get('flag')
default_param = '..."
if(experiment_value):
  param = statsig.get_experiment('autotune').get('param_name')
else:
  param = default_param

# use param in code

Start this experiment at the same time you launch your autotune.