Bayesian Experiments

How Bayesian testing works in Statsig

Experiments are frequentist by default. To switch to Bayesian mode, go to Advanced Settings.

You can't modify the experiment type after the experiment starts.

Deep-dive analysis reflects Bayesian statistics.

Informed Bayesian

Bayesian experiments let you specify a prior belief on the relative average treatment effect. Statsig combines the prior distribution with the observed data to display prior-adjusted results. To enable this, select the "use informative priors" option.

Drawing the correct prior distribution from historical data

When using Bayesian with informative priors, your organization must understand what influence the priors have over your experimental results and must have established a reliable prior based on domain knowledge. Use these patterns to derive priors:

1. You can use the $AVG(\text{average treatment effect})$ of past experiments with a similar setup and population as your prior mean. You can use the standard deviation, or a multiple of it, as the prior standard error.
2. You can also use the $AVG(\text{observed standard error})$ as your prior standard error.

Implementation details

Denote $\mathcal{N}(ATE_{prior}, STE_{prior}^2)$ as the prior distribution, where $ATE_{prior}$ is the average treatment effect and $STE_{prior}$ is the standard error. Similarly, $\mathcal{N}(ATE_{observed}, STE_{observed}^2)$ as the observed distribution.

The posterior distribution is then calculated as

$$

ATE_{post} = \frac{ \frac{ATE_{prior}}{STE_{prior}^2} + \frac{ATE_{observed}}{STE_{observed}^2} }{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} } $$

$$

STE_{post}^2 = \frac{1}{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} } $$

If the prior is not specified, the $\mathcal{N}(ATE_{prior}, STE_{prior}^2)$ is represented as $\mathcal{N}(0, \infty)$.

Bayesian statistics glossary

Bayesian A/B tests use different terminology from the frequentist framework. These terms are often more intuitive for communicating results to non-technical audiences.