Bayesian Experiments

Bayesian Testing in Statsig

Experiments are frequentist by default. To switch to Bayesian mode, go to Advanced Settings.

The experiment type cannot be modified once the experiment starts.

Deep dive analysis should also reflect Bayesian statistics

Informed Bayesian

Bayesian experiments allow you to specify a prior belief on the relative average treatment effect. Statsig will combine the prior distribution with the observed data to display the prior-adjusted results. You can enable this by selecting the option to "use informative priors".

Implementation Details

Denote $\mathcal{N}(ATE_{prior}, STE_{prior}^2)$ as the prior distribution, where $ATE_{prior}$ is the average treatment effect and $STE_{prior}$ is the standard error. Similarly, $\mathcal{N}(ATE_{observed}, STE_{observed}^2)$ as the observed distribution.

The posterior distribution is then calculated as

$$\LARGE ATE_{post} = \frac{ \frac{ATE_{prior}}{STE_{prior}^2} + \frac{ATE_{observed}}{STE_{observed}^2} }{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} }$$ $$\LARGE STE_{post}^2 = \frac{1}{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} }$$

If the prior is not specified, the $\mathcal{N}(ATE_{prior}, STE_{prior}^2)$ is represented as $\mathcal{N}(0, \infty)$.

Bayesian Statistics

Bayesian A/B tests have a glossary that are different from the frequentist framework and often believed to be more intuitive in communication to non-technical audience.

• Credible Interval: the interval which we believe contains the true parameter at the given probability
• Chance to Beat: the probability that the test is better than control
• Expected Loss: the average potential risk if you ship test