Bandit FAQs

When should I see data show up in a bandit?

Diagnostic data appears in near-real time in the logstream on your bandit.

Data in the models/results section depends on your settings. Statsig doesn't count units as enrolled until their attribution window has elapsed. For example, if you have a 24-hour attribution window, it takes 24 hours to see your first units and model updates. Statsig trains models hourly with a few minutes of data delay, so there may be 1 to 2 additional hours of delay beyond that period.

This operation doesn't have a strict SLA on Statsig's side. Delays may occur due to data quality checks or other investigations to prevent bad model training data from affecting your production environment.

What do I put into the Variant JSON?

The variants return this JSON as a config, similar to dynamic configs or experiments. You can put an identifier for the variant, or specify a set of attributes that will be accessed in code to avoid writing conditional statements to fetch the corresponding attributes.

Why use a linked experiment?

Bandits optimize for a single target metric or event, which alone doesn't provide a detailed analysis of how the bandit affects your users. Wrapping a bandit in an experiment (where the control group receives a default experience and the variant group receives the bandit's experience) lets you get deeper insights on its impact on end users. This step is highly recommended.

Where is my Autotune data available?

Statsig downloads Autotune data hourly into the statsig_first_exposures metric source. Select the history modal on the autotune to view the SQL and tables generated for an individual autotune.

Why was a "losing" variant chosen?

Autotune assumes that a metric is consistent over time. A metric with strong seasonality (for example, a metric that drops on weekends) isn't a good candidate for an Autotune experiment. With seasonality, the following situation is possible:

• A variant that consistently performs worse than others receives less and less traffic until it has essentially zero traffic.
• Other variants, still receiving traffic, see their metric drop when seasonal effects occur.
• The variant with zero traffic doesn't see that metric drop (because no traffic is allocated to it), which causes Statsig to select it as the winning variant.

This can sometimes cause Statsig to choose a "losing" variant as the winner.