Confidence Intervals

How Statsig calculates confidence intervals for experiment metrics, including the formulas, assumptions, and how to interpret intervals in scorecards.

A confidence interval is the range of metric deltas consistent with your experiment data at a chosen confidence level. Statsig draws it as the gray, red, or green bar on each Pulse metric; a 95% interval that excludes zero is statistically significant at $\alpha = 0.05$ . Read the interval, not only the p-value, when you need to know how large the effect could plausibly be.

A 95% confidence interval contains the true effect 95% of the time: if you ran an experiment 100 times, the true metric delta would fall inside the interval about 95 times. If the true effect is zero, you expect the interval to exclude zero only 5% of the time (a false positive). A wider interval means less certainty about the exact size of the effect.

Confidence interval visualization showing statistical significance

Computing confidence intervals

Statsig calculates confidence intervals with a two-sample z-test. The test requires the variance of the metric delta, which Statsig derives differently for each metric type; refer to standard error and mean variance. After establishing the variance of the delta, Statsig computes the confidence interval.

Two-sided tests

For the absolute metric delta, Statsig computes the confidence interval as:

CI(\Delta \overline{X}) = \Delta \overline{X} \pm Z_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}

where:

$Z_{\alpha/2}$ is the z-critical value for the significance level you want (1.96 for the standard $\alpha=0.05$ and 95% confidence interval) for a two-sided test.
$var(\Delta \overline{X})$ is the variance of the absolute delta.

The confidence interval for the relative metric delta uses one of two methods: Fieller Intervals or the Delta Method. You can choose either method. Statsig turns on Fieller Intervals by default for new customers.

With Fieller Intervals, the relative metric delta confidence interval is:

CI(\% \Delta \overline{X} ) = \frac{1}{1-g} ( \frac{\overline{X_T}}{\overline{X_C}} - 1 \pm \frac{Z_{\alpha/2}}{\sqrt{n_C} \cdot \overline{X_C}} \sqrt{(1-g) \cdot \frac{var(X_T)}{n_T(n_T-1)} + \frac{\overline{X_T} var(X_C)}{\overline{X_C} n_C (n_C-1)}})

With the Delta Method, the confidence interval is:

\begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}}\\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot\sqrt{(\frac{\overline X_t}{\overline X_c})^{2} \cdot (\frac{var(X_c)}{n_c \cdot \overline X_c^2} + \frac{var(X_t)}{n_t \cdot \overline X_t^2})} \cdot 100\% \end{split}

If you use the Delta Method and the control mean isn't significantly different from zero, the interval simplifies to:

\begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}} \\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot \frac{\sqrt{{var\left(\Delta \overline X\right)}}}{\overline X_c} \cdot 100\% \end{split}

A statistically significant p-value and a relative delta confidence interval that excludes zero don't always align. The p-value of the absolute difference between test and control can be significant while uncertainty in the control mean affects the relative delta interval. With the Delta Method, the relative delta confidence interval may cross zero; with Fieller Intervals, it may appear as a point estimate.

One-sided tests

For one-sided tests, the confidence interval calculation changes to redistribute the false positive rate toward the direction you're testing, either increases or decreases in the metric:

CI(\Delta \overline{X}) = \begin{cases} \left[\Delta \overline{X} - Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}}, \quad +\infty \right) & \text{if right-hand test}\\ \\ \left(-\infty, \quad \Delta \overline{X} + Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}} : \right] & \text{if left-hand test} \end{cases}

where:

$Z_{\alpha}$ is the z-critical value for the significance level you want (1.645 for the standard $\alpha=0.05$ and 95% confidence interval) for a one-sided test.
$var(\Delta \overline{X})$ is the variance of the absolute delta.
The interval Statsig uses depends on whether the one-sided test looks for increases or decreases in the metric.

Welch's t-test for small sample sizes

For small sample sizes, Statsig uses Welch's t-test instead of a z-test. Welch's t-test handles samples of unequal size or variance without increasing the false positive rate. The confidence interval has the same form as the two-sample z-test interval for one- and two-sided tests, with the t-critical value with $\nu$ degrees of freedom in place of the z-critical value.

For a two-sided test, the confidence interval is:

CI(\Delta \overline{X}) = \Delta \overline{X} \pm t_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}

\nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} = \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}

where $N_t$ and $N_c$ are the number of users in the test and control groups. For a large number of degrees of freedom, the t-statistic converges with the z-statistic, so Statsig uses Welch's t-test only when $\nu < 100$ .

Comparing experiment data to a fixed baseline with a one-sample t-test

To answer a question such as "Does my test variant lead to a click-through rate higher than 0.5?", define a fixed-baseline comparison when you add metrics to the experiment. For details, refer to one-sample tests.

Statsig calculates the confidence interval as:

CI(\Delta \overline X) = (\overline X_{group} - fixed \ value) \pm Z \cdot\sqrt{{var( \overline X_{group})}}

Was this helpful?