# Standard Error & Mean Variance

The confidence interval and p-value of a metric delta between two groups both depend on the standard error of the mean of each group. The standard error is sometimes denoted "SE" or "std err". You obtain the standard error of the mean by dividing the sample standard deviation of $X$ by the square root of the number of users in the group.

$$
\sigma\_\{\overline X} = \frac\{\sigma\_\{X}}\{\sqrt\{N}} = \sqrt\{\frac\{var(X)}\{N}} = \sqrt\{var(\overline\{X})}
$$

Standard deviation is the square root of variance. Because variances are easier to manipulate algebraically, Statsig derives the variance for each metric type and then takes the square root to obtain the confidence intervals.

Pulse shows the standard error of the mean of each group alongside the units and mean of each group.

## Computing variance

The variance of the absolute metric delta is the sum of the variances of the test and control means:

$$
var(\Delta \overline X) =var(\overline X\_t - \overline X\_c) = var(\overline X\_t) + var(\overline X\_c)
$$

Computing this correctly requires accurate variance of the means for each group.

### Count and sum metrics

For count and sum metrics, Statsig obtains the variance of the sample mean for a given group directly from the sample variance:

$$
var(\overline\{X}) = \frac\{var(X)}\{N} = \frac\{\frac\{1}\{N-1}\sum\_\{i=0}^\{N}(X\_i-\overline\{X})^2}\{N}
$$

Where:

* $N$ is the number of users in the group
* $X\_i$ is the metric value for user $i$
* $\overline\{X}$ is the user-level average of $X$ for users in that group

### Ratio and mean metrics

Ratio and mean metrics combine multiple variables $X$ and $Y$ rather than a single variable $X$. The variance of these metrics depends on both the numerator and denominator variables, which are typically correlated. For metric $R$, the group mean $\overline\{R}$ and group variance of the mean $var(\overline\{R})$ must account for this correlation.

For example, in a *clicks per session* metric, the number of clicks and the number of sessions come from the same group of users and aren't independent.

To account for any correlation, Statsig computes the variance of the mean of a ratio metric $R$ using the delta method:

$$
var(\overline R) = var\left(\frac\{\overline X}\{\overline Y}\right)
:= \left(\frac\{\overline X}\{\overline Y}\right)^2 \cdot \left(\frac\{var(\overline X)}\{\overline X^2} + \frac\{var(\overline Y)}\{\overline Y^2} - 2 \cdot \frac\{covar(\overline X, \overline Y)}\{\overline X\cdot \overline Y} \right)
$$

where Statsig computes the variance of the numerator and denominator means in the same way as detailed above for count metrics, and the covariance is

$$
covar(\overline X, \overline Y) = \frac\{covar(X, Y)}\{N} = \frac\{\frac\{1}\{N-1}\sum\_\{i=0}^\{N}(X\_i-\overline X)\cdot (Y\_i-\overline Y)}\{N}
$$