Standard Error & Mean Variance

How Statsig computes variance for experiment metrics, including handling of ratio metrics, clustered data, and user-level aggregation across exposures.

The standard error of a group's mean measures how much that mean would vary across repeated samples. Statsig uses it to build the confidence interval and p-value for every metric delta. It equals the sample standard deviation of $X$ divided by the square root of the number of users in the group, and Pulse shows it as Std Err next to each group's units and mean. This page gives the variance formula for each metric type.

\sigma_{\overline X} = \frac{\sigma_{X}}{\sqrt{N}} = \sqrt{\frac{var(X)}{N}} = \sqrt{var(\overline{X})}

Standard deviation is the square root of the variance. Because variances are easier to manipulate algebraically, Statsig derives the variance for each metric type and then takes the square root to obtain the confidence intervals.

Computing variance

The variance of the absolute metric delta is the sum of the variances of the test and control means:

var(\Delta \overline X) =var(\overline X_t - \overline X_c) = var(\overline X_t) + var(\overline X_c)

The task is therefore to compute the variance of the mean for each group.

Count and sum metrics

For count and sum metrics, Statsig derives the variance of the sample mean for a group directly from the sample variance:

var(\overline{X}) = \frac{var(X)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline{X})^2}{N}

where:

$N$ is the number of users in the group.
$X_i$ is the metric value for user $i$ .
$\overline{X}$ is the user-level average of $X$ for users in that group.

Ratio and mean metrics

Ratio and mean metrics combine two variables, a numerator $X$ and a denominator $Y$ , which are typically correlated because both come from the same users. For example, in a clicks per session metric, clicks and sessions are two sets of observations from the same group of users, so they aren't independent. The metric $R$ has a group mean $\overline{R}$ and a group variance of the mean $var(\overline{R})$ .

To account for this correlation, Statsig obtains the variance of the mean of a ratio metric $R$ with the delta method:

var(\overline R) = var\left(\frac{\overline X}{\overline Y}\right) := \left(\frac{\overline X}{\overline Y}\right)^2 \cdot \left(\frac{var(\overline X)}{\overline X^2} + \frac{var(\overline Y)}{\overline Y^2} - 2 \cdot \frac{covar(\overline X, \overline Y)}{\overline X\cdot \overline Y} \right)

where Statsig computes the variance of the numerator and denominator means with the same formula as for count and sum metrics, and the covariance is:

covar(\overline X, \overline Y) = \frac{covar(X, Y)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline X)\cdot (Y_i-\overline Y)}{N}

Was this helpful?