On this page

For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.

CUPED

How Statsig uses CUPED variance reduction to improve experiment sensitivity by adjusting for pre-experiment user behavior on metric values.

CUPED (Controlled-experiment Using Pre-Existing Data) reduces a metric's variance by adjusting each user's metric value with that user's pre-exposure data. Pre-exposure data is the same metric measured over the 7 days before the user's exposure. Statsig defines this window per user, not as a fixed window before the experiment starts for all users. The adjustment increases confidence in experiment metrics and reduces pre-exposure bias, for example when groups were randomly different before you applied any treatment. Use CUPED when a metric is autocorrelated over time. Use CURE, which extends CUPED on Warehouse Native, when you need covariates beyond the metric's own pre-exposure history, such as in new-user experiments or for metrics that aren't autocorrelated.

Statsig Cloud always uses the 7-day pre-exposure window. On Warehouse Native, Statsig recommends the 7-day window, but you can set any length. For how CUPED fits with Statsig's other variance reduction options, refer to Variance reduction.

CUPED for simple aggregations

The original Microsoft paper describes the methodology for simple aggregations, as does Statsig's in-depth CUPED article.

Statsig Cloud uses stratification alongside CUPED to account for users who have no pre-exposure data. Statsig groups users into strata based on the pre-exposure data available, estimates treatment and control effects within each stratum, then aggregates the strata to produce an overall result. Statsig then applies the standard difference-in-means and variance estimation. This approach retains users with missing pre-exposure data while still reducing variance where pre-exposure data exists.

CUPED for ratio metrics

The Microsoft paper also describes how to implement CUPED for metrics with a different analysis unit (Appendix B). Statsig extends this methodology to ratio metrics, where a numerator and a denominator represent each experiment unit. The variance reduction process finds the variance of the experiment data, the variance of the pre-exposure data, and the covariance between the two.

Denote the numerator, denominator, pre-exposure numerator, and pre-exposure denominator of a unit as YY, NN, XX, and MM, respectively. Using the CUPED-reduced variance formula,

Var(YcvNcv)=Var(YN)+θ2Var(XM)−2θCov(YN,XM)Var(\frac{Y_{cv}}{N_{cv}})=Var(\frac{Y}{N})+\theta^2 Var(\frac{X}{M})-2\theta Cov(\frac{Y}{N}, \frac{X}{M})

where Statsig finds optimal θ\theta as

Cov(YN,XM)Var(XM)\frac{Cov(\frac{Y}{N}, \frac{X}{M})}{Var(\frac{X}{M})}

expanded to

Cov(YμN−μYNμN2,XμM−μXMμM2)Var(XμM−μXMμM2)\frac{Cov(\frac{Y}{\mu_N}-\frac{\mu_Y N}{\mu^2_N}, \frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})}{Var(\frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})}

This gives:

Yc^Nc^=YcNc−θ(XcMc−E[R])\frac{\hat{Y_{c}}}{\hat{N_{c}}}=\frac{Y_{c}}{N_{c}}-\theta( \frac{X_{c}}{M_{c}} - \mathbb{E}[R])
Yt^Nt^=YtNt−θ(XtMt−E[R])\frac{\hat{Y_{t}}}{\hat{N_{t}}}=\frac{Y_{t}}{N_{t}}-\theta( \frac{X_{t}}{M_{t}} - \mathbb{E}[R])

Because E[R]\mathbb{E}[R] is hard to derive, the expectation term is the same for both groups. Substituting E[R]\mathbb{E}[R] with XcMc\frac{X_{c}}{M_{c}} transforms these two formulas into the following:

Ycv(control)Ncv(control)=Y(control)N(control)\frac{Y_{cv}(control)}{N_{cv}(control)}=\frac{Y(control)}{N(control)}
Ycv(test)Ncv(test):=Y(control)N(control)−(Y(control)N(control)−θX(control)M(control))+(Y(test)N(test)−θX(test)M(test)):=Y(test)N(test)−θX(test)M(test)+θX(control)M(control)\frac{Y_{cv}(test)}{N_{cv}(test)} :=\frac{Y(control)}{N(control)} - (\frac{Y(control)}{N(control)} - \theta \frac{X(control)}{M(control)}) + (\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)}) :=\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)} + \theta \frac{X(control)}{M(control)}

Using the optimal θ\theta, Statsig reduces group-level variance by applying the parameter to calculate the adjustment. An across-group θ\theta doesn't necessarily reduce the variance for one group, or the sum of variances of all groups, but in most cases it does. Simulations show that 98.3% of metrics saw a decrease through CUPED.

Statsig applies the CUPED-reduced variance when all of the following are true:

  • The core assumptions of the CUPED model hold. Rounding error or other data artifacts can violate them:
    • E(X^)=E(X)E(\hat{X}) = E(X)
    • The pooled variance of the adjusted population across groups is less than the variance of the unadjusted population.
  • More than 100 units have pre-exposure values.
  • More than 5% of units have pre-exposure values.

Was this helpful?