Delta Method
How Statsig Warehouse Native applies the delta method to compute variances for ratio metrics and other non-linear functions of user-level data.
Delta method for ratio metrics
Statsig uses the delta method when calculating the variance for variables that have a numerator and denominator.
The variance of ratio and mean metrics depends on the numerator and denominator variables, which are typically correlated. For example, consider a clicks per session metric. The number of clicks and the number of sessions are two sets of observations from the same group of users, so they aren't independent of each other. To account for this correlation, Statsig obtains the variance of a ratio metric R using the delta method:
where the variance of the numerator and denominator means are computed as described for count metrics, and the covariance is
Delta method for relative lifts
Statsig may also use the delta method when calculating the confidence interval for relative lifts. The other methodology is Fieller Intervals. The delta method is a heuristic for Fieller Intervals that converges with a large population.Was this helpful?