Skip to main content

​
Bayesian Testing in Statsig

Experiments are frequentist by default. To switch to Bayesian mode, go to Advanced Settings.
Bayesian experiment configuration interface
The experiment type cannot be modified once the experiment starts.
Bayesian experiment type selection interface
Deep dive analysis should also reflect Bayesian statistics
Bayesian deep dive analysis interface

​
Informed Bayesian

Bayesian experiments allow you to specify a prior belief on the relative average treatment effect. Statsig will combine the prior distribution with the observed data to display the prior-adjusted results. You can enable this by selecting the option to “use informative priors”.
Informative priors configuration interface

​
Drawing the Correct Prior Distribution From Historical Data

If you are using the Bayesian with informative priors, the assumption is that you have a clear understanding of what power the priors have over your experimental results, and your organization has established a reliable prior based on the domain knowledge. With that said, here are some patterns people follow to derive their priors:
  1. You can use the AVG(average treatment effect)AVG(\text{average treatment effect}) of past experiments with a similar setup and population as your prior mean. You can use the standard deviation, or a multiple of it, as the prior standard error.
  2. You can also use the AVG(observed standard error)AVG(\text{observed standard error}) as your prior standard error.

​
Implementation Details

Denote N(ATEprior,STEprior2)\mathcal{N}(ATE_{prior}, STE_{prior}^2) as the prior distribution, where ATEpriorATE_{prior} is the average treatment effect and STEpriorSTE_{prior} is the standard error. Similarly, N(ATEobserved,STEobserved2)\mathcal{N}(ATE_{observed}, STE_{observed}^2) as the observed distribution. The posterior distribution is then calculated as ATEpost=ATEpriorSTEprior2+ATEobservedSTEobserved21STEprior2+1STEobserved2 ATE_{post} = \frac{ \frac{ATE_{prior}}{STE_{prior}^2} + \frac{ATE_{observed}}{STE_{observed}^2} }{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} } STEpost2=11STEprior2+1STEobserved2 STE_{post}^2 = \frac{1}{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} } If the prior is not specified, the N(ATEprior,STEprior2)\mathcal{N}(ATE_{prior}, STE_{prior}^2) is represented as N(0,∞)\mathcal{N}(0, \infty).

​
Bayesian Statistics

Bayesian A/B tests have a glossary that are different from the frequentist framework and often believed to be more intuitive in communication to non-technical audience.
  • Credible Interval: the interval which we believe contains the true parameter at the given probability
  • Chance to Beat: the probability that the test is better than control
  • Expected Loss: the average potential risk if you ship test
⌘I