Skip to main content

Standard Error

The standard error (sometimes denoted "SE" or "std err") of the mean of each group is required for computing the confidence interval and p-value of a metric delta between those groups. The standard error of the mean can be obtained by dividing the sample standard deviation of XX by the square root of the number of users in the group.

σX‾=σXN=var(X)N\sigma_{\overline X} = \frac{\sigma_{X}}{\sqrt{N}} = \sqrt{\frac{var(X)}{N}}

Note that standard deviation is the square root of the variance. Since variances are easier to manipulate algebraically, here we derive the variance for each metric type and then take the square root to obtain the confidence intervals.

Pulse displays the standard error of the mean of each group alongside the units and mean of each group.

Computing Variance​

The variance of the absolute metric delta is simply the sum of the variances of the test and control means:

var(ΔX‾)=var(X‾t−X‾c)=var(X‾t)+var(X‾c)var(\Delta \overline X) =var(\overline X_t - \overline X_c) = var(\overline X_t) + var(\overline X_c)

In other words, it comes down to correctly calculating the variance of the means for each group.

Count and Sum Metrics​

For count and sum metrics, the variance of the sample mean for a given group is obtained directly from the sample variance:

var(X‾)=var(X)N=1N−1∑i=0N(Xi−X‾)2Nvar(\overline{X}) = \frac{var(X)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline{X})^2}{N}

Where:

  • NN is the number of users in the group
  • XiX_i is the metric value for user ii
  • X‾\overline{X} is the user-level average of XX for users in that group

Ratio and Mean Metrics​

Some metrics like ratio and mean metrics combines multiple variables XX and YY instead of a single variable XX. The variance of these metrics depends upon both the numerator and denominator variables, which are typically correlated. We'll call the metric of interest RR and we can compute the group mean R‾\overline{R} and group variance of the mean var(R‾var(\overline{R}).

For example, consider a clicks per session metric. The number of clicks and the number of sessions are two sets of observations coming from the same group of users, so they are not independent from each other.

To properly account for any correlation, the variance of the mean of a ratio metric RR is obtained using the delta method:

var(R‾)=var(X‾Y‾)=(X‾Y‾)2⋅(var(X‾)X‾2+var(Y‾)Y‾2−2⋅covar(X‾,Y‾)X‾⋅Y‾)var(\overline R) = var\left(\frac{\overline X}{\overline Y}\right) = \left(\frac{\overline X}{\overline Y}\right)^2 \cdot \left(\frac{var(\overline X)}{\overline X^2} + \frac{var(\overline Y)}{\overline Y^2} - 2 \cdot \frac{covar(\overline X, \overline Y)}{\overline X\cdot \overline Y} \right)

where the variance of the numerator and denominator means are computed in the same way as detailed above for count metrics, and the covariance is

covar(X‾,Y‾)=covar(X,Y)N=1N−1∑i=0N(Xi−X‾)⋅(Yi−Y‾)Ncovar(\overline X, \overline Y) = \frac{covar(X, Y)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline X)\cdot (Y_i-\overline Y)}{N}