Confidence Intervals

Confidence intervals are an intuitive way to quantify the uncertainty in the observed metric deltas. A 95% confidence interval should contain the true effect 95% of the time. This means that if we ran an experiment 100 times, the true value of the metric delta should be inside the confidence intervals 95 times.

In practical terms, a 95% confidence interval that doesn't contain zero (the green bar above) represents a statistically significant result (with α = 0.05). Only 5% of the time would we expect to see the confidence interval exclude zero if the true effect was zero (a.k.a. a false positive). Larger confidence intervals imply less certainty in the exact size of the effect with a larger range of likely values.

Computing Confidence Intervals

Confidence intervals in Statsig are calculated using a two-sample z-test. This test requires knowledge of the variance in the metric delta we're measuring, which is derived differently depending on the type of metric (details here).

Once we've established the variance of the delta, it's straightforward to compute the confidence intervals.

Two-Sided Tests

For the absolute metric delta, the confidence interval is given by:

$$\LARGE CI(\Delta \overline{X}) = \Delta \overline{X} \pm Z_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}$$

where:

• $Z_{\alpha/2}$ is the z-critical value for the desired significance level (1.96 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a two-sided test
• $var(\Delta \overline{X})$ is the variance of the absolute delta (details here)

The confidence interval for the relative metric delta can use one of two methods, Fieller Intervals and the Delta Method. While customers can opt for either method to be used in their Statsig console, Statsig recommends and automatically opts-in all new customers to use Fieller Intervals by default.

When using Fieller Intervals, the relative metric delta CI can be computed using:

$$\Large CI(\% \Delta \overline{X} ) = \frac{1}{1-g} ( \frac{\overline{X_T}}{\overline{X_C}} - 1 \pm \frac{Z_{\alpha/2}}{\sqrt{n_C} \cdot \overline{X_C}} \sqrt{(1-g) \cdot \frac{var(X_T)}{n_T(n_T-1)} + \frac{\overline{X_T} var(X_C)}{\overline{X_C} n_C (n_C-1)}})$$

When using the Delta Method, the confidence interval is:

$$\Large \begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}}\\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot\sqrt{(\frac{\overline X_t}{\overline X_c})^{2} \cdot (\frac{var(X_c)}{n_c \cdot \overline X_c^2} + \frac{var(X_t)}{n_t \cdot \overline X_t^2})} \cdot 100\% \end{split}$$

If using the Delta Method and the the control mean is not significantly away from zero, then it's simplified to:

$$\Large \begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}} \\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot \frac{\sqrt{{var\left(\Delta \overline X\right)}}}{\overline X_c} \cdot 100\% \end{split}$$

One-Sided Tests

When running one-sided tests, the form of the confidence interval calculation changes slightly to account for a redistribution of desired false positive rate when looking for increases or decreases in the metric:

$$\Large CI(\Delta \overline{X}) = \begin{cases} \left[\Delta \overline{X} - Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}}, \quad +\infty \right) & \text{if right-hand test}\\ \\ \left(-\infty, \quad \Delta \overline{X} + Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}} \: \right] & \text{if left-hand test} \end{cases}$$

where:

• $Z_{\alpha}$ is the z-critical value for the desired significance level (1.645 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a one-sided test
• $var(\Delta \overline{X})$ is the same as for two-sided tests
• the choice of confidence interval depends on if the one-sided test is looking for increases or decreases in the metric

Welch's T-test for Small Sample Sizes

For small sample sizes, we use Welch's t-test instead of a standard z-test. This statistical test is a better choice for handling samples of unequal size or variance without increasing the false positive rate. The structure of the confidence interval calculation remains the same as above (depending on 1- or 2-sided test), replacing the z-critical value with the t-critical value with degrees of freedom $\nu$.

For a two-sided test, the confidence interval is therefore:

$$\Large CI(\Delta \overline{X}) = \Delta \overline{X} \pm t_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}$$ $$\Large \nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} = \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}$$

Where $N_t$ and $N_c$ are the number of users in the test and control groups, respectively. Note that for large number of degrees of freedom, the t-statistic converges with the z-statistic. Therefore, Welch's t-test is used only when $\nu < 100$.

Comparing Experiment Data to a Fixed Baseline: One-sample T-test

Sometimes we want to answer questions like "Does my test variant lead to a click through rate higher than 0.5?". You can define a fixed-baseline comparison when adding metrics to the experiment.

The confidence interval is calculated by

$$\Large CI(\Delta \overline X) = (\overline X_{group} - fixed \ value) \pm Z \cdot\sqrt{{var( \overline X_{group})}}$$