Delta Method

The delta method approximates the variance of a ratio metric, such as clicks per session, from the variances of its numerator and denominator and the covariance between them. Statsig uses it because the numerator and denominator come from the same users and are correlated. For confidence intervals on relative lifts, Statsig recommends Fieller intervals instead; the delta method is the large-sample approximation to the Fieller interval.

Delta method for ratio metrics

Statsig uses the delta method to calculate the variance of metrics that have a numerator and a denominator.

The variance of ratio and mean metrics depends on the numerator and denominator variables, which are typically correlated. For example, consider a clicks per session metric. The number of clicks and the number of sessions are two sets of observations from the same group of users, so they aren't independent of each other. To account for this correlation, Statsig obtains the variance of a ratio metric $R$ using the delta method:

Statsig computes the variance of the numerator and denominator means the same way it does for count metrics (refer to Variance). The covariance is

Delta method for relative lifts

Statsig can also use the delta method to calculate the confidence interval for relative lifts. The alternative, which Statsig recommends, is Fieller intervals. The delta method is an approximation of the Fieller interval that converges to it as the sample size grows.