Introduction

The Statsig Stats Engine is the set of statistical calculations behind the experiment results you read in Pulse. It turns exposure and metric data into a metric delta, confidence interval, and p-value for each metric, applies variance reduction, and supports frequentist, SPRT, and Bayesian analysis methods. Use these pages when you need the formula or assumption behind a number in your results.

What the Stats Engine calculates and how

• Metric deltas: The difference in mean metric value between the test and control groups, in absolute and relative terms.
• Standard error and mean variance: How much each group's mean would vary across repeated samples. This is the input to every confidence interval and p-value.
• Confidence intervals: The gray, red, and green bars in Pulse, which represent the uncertainty in each metric delta.
• p-values: The probability of observing a metric delta at least as extreme as the one measured, if the treatment has no effect on that metric.
• Topline and projected impact: The daily effect of the experiment on a company metric, and the estimated effect if you launch the test group to all users.
• Variance reduction: The techniques Statsig applies so the same effect reaches significance with fewer users.
• CUPED: Uses each user's pre-exposure data to reduce variance. Statsig applies it automatically.
• Pre-experiment bias: How Statsig detects groups that already differed before exposure and flags the affected metrics.
• Frequentist sequential testing: Adjusts p-values and confidence intervals so you can read results early without inflating false positives.
• Sequential probability ratio tests: Compares the likelihood of your data under the alternative and null hypotheses, and can accept either one.
• Bayesian experiments: Reports chance to beat control and expected loss, with optional informative priors.