Monitoring your Contextual MAB

There are three primary ways to monitor autotune performance.

Linked experiments

The most reliable way to evaluate whether a bandit is working is to measure whether it drives more of the targeted behavior compared to a baseline experience. You can set up and link an A/B test in Statsig to evaluate this, which also lets you monitor other user behaviors and guardrail metrics.

This is the gold standard of measurement and is highly encouraged.

Standard practice is to wrap the autotune in an experiment with a binary parameter, either as 50/50 or a 90/10 holdback. You can link the experiment to the autotune to get the results on the autotune page. In code, this might look like:

experiment_value = statsig.get_experiment('wrapping_experiment').get('flag')
default_param = '..."
if(experiment_value):
  param = statsig.get_experiment('autotune').get('param_name')
else:
  param = default_param

# use param in code

You would start this experiment at the same time that you launch your autotune.

Success rate

Statsig tracks the cumulative and daily success rate of your variants over time. This can be tricky to interpret. For example, variant A may have lower CTR, but the users being served variant A might have had even lower CTR on other variants. Use this view for tracking and understanding, and to identify outlier variants with notably high or low performance.

Traffic allocation

Traffic allocation shows where Statsig is sending users who see your Autotune. Use this view to identify whether a variant is dominating traffic or receiving no traffic.

Model features

Statsig tracks and surfaces coefficients and feature importance; this can be very useful for understanding which features might be worth further study, or which populations may have unmet needs in your product.

• Importance is an estimate of the influence of a feature on the outcome: how much the feature contributes to the prediction.
• A positive coefficient means that feature leads to an outcome being more likely (or for continuous outcome spaces is associated with a higher outcome). A negative coefficient means the outcome is less likely, or is associated with a lower continuous outcome.