Skip to main content

Standard Error

The standard error of the means is required for computing the confidence interval and p-value of a metric delta. It can be obtained by dividing the sample standard deviation of X by the square root of the number of users in the group.

σX‾=σXN=var(X)N\sigma_{\overline X} = \frac{\sigma_{X}}{\sqrt{N}} = \sqrt{\frac{var(X)}{N}}

Note that standard deviation is the square root of the variance. Since variances are easier to manipulate algebraically, here we derive the variance for each metric type and then take the square root to obtain the confidence intervals.

Computing Variance​

The variance of the absolute metric delta is simply the sum of the variances of the test and control means:

var(ΔX‾)=var(X‾t−X‾c)=var(X‾t)+var(X‾c)var(\Delta \overline X) =var(\overline X_t - \overline X_c) = var(\overline X_t) + var(\overline X_c)

In other words, it comes down to correctly calculating the variance of the means for each group.

Count and Sum Metrics​

For count and sum metrics, the variance of the sample mean for a given group is obtained directly from the sample variance:

var(X‾)=var(X)N=1N−1∑i=0N(Xi−X‾)2Nvar(\overline X) = \frac{var(X)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline X)^2}{N}

Where:

  • N is the number of users in the group
  • Xi is the metric value for user i
  • X-bar is the user-level average of X for users in that group

Ratio and Mean Metrics​

The variance of ratio and mean metrics depends upon the numerator and denominator variables, which are typically correlated. For example, consider a clicks per session metric. The number of clicks and the number of sessions are two sets of observations coming from the same group of users, so they are not independent from each other. To properly account for this correlation, the variance of the mean of a ratio metric R is obtained using the delta method:

var(R‾)=var(X‾Y‾)=(X‾Y‾)2⋅(var(X‾)X‾2+var(Y‾)Y‾2−2⋅covar(X‾,Y‾)X‾⋅Y‾)var(\overline R) = var\left(\frac{\overline X}{\overline Y}\right) = \left(\frac{\overline X}{\overline Y}\right)^2 \cdot \left(\frac{var(\overline X)}{\overline X^2} + \frac{var(\overline Y)}{\overline Y^2} - 2 \cdot \frac{covar(\overline X, \overline Y)}{\overline X\cdot \overline Y} \right)

where the variance of the numerator and denominator means are computed in the same way as detailed above for count metrics, and the covariance is

covar(X‾,Y‾)=covar(X,Y)N=1N−1∑i=0N(Xi−X‾)⋅(Yi−Y‾)Ncovar(\overline X, \overline Y) = \frac{covar(X, Y)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline X)\cdot (Y_i-\overline Y)}{N}