Skip to main content

​
CUPED - Controlled-experiment Using Pre-Existing Data

CUPED (short for Controlled-experiment Using Pre-Existing Data) is a technique which leverages user information from before an experiment to reduce the variance, and increase confidence in experimental metrics. At Statsig, this pre-experiment data is defined as the 7 days before each user’s exposure rather than a fixed window before the experiment starts for all users. This can help to debias experiments which have meaningful pre-exposure bias (e.g. the groups were randomly different before any treatment was applied). Our Cloud product uses a 7-day window for CUPED calculation. For Warehouse Native customers, a 7-day window is recommended, but you have the flexibility to customize it to any length. See more at the Variance Reduction page. To dive deep into the methodology, see CURE by Statsig.

​
CUPED for Simple Aggregations

The methodology for simple aggregations is described in the original Microsoft paper, as well as our in-depth article on the technique. The Cloud product uses stratification alongside CUPED to account for users who may not have pre-experiment data. Users are grouped into strata based on available pre-experimentation information. Treatment and control effects are first estimated within each stratum, then aggregated to produce an overall result. We then apply the standard difference-in-means and variance estimation. This approach allows us to retain users with missing pre-data while still benefiting from variance reduction where applicable.

​
CUPED for Ratio Metrics

The Microsoft paper also gives details on how to implement CUPED for those with a different analysis unit (Appendix B). On Statsig, we extend it to work for our ratio metrics, where each experiment unit is represented by a numerator and a denominator. The variance reduction process is performed by finding the variance of experiment data, pre-experiment data, and the covariance between the two. Denote the numerator, denominator, pre-experiment numerator, and pre-experiment denominator of a unit as YY, NN, XX, and MM, respectively. Using the CUPED-reduced variance formula, Var(YcvNcv)=Var(YN)+θ2Var(XM)−2θCov(YN,XM) Var(\frac{Y_{cv}}{N_{cv}})=Var(\frac{Y}{N})+\theta^2 Var(\frac{X}{M})-2\theta Cov(\frac{Y}{N}, \frac{X}{M}) where optimal θ\theta is found as Cov(YN,XM)Var(XM) \frac{Cov(\frac{Y}{N}, \frac{X}{M})}{Var(\frac{X}{M})} expanded to \ Cov(YμN−μYNμN2,XμM−μXMμM2)Var(XμM−μXMμM2) \frac{Cov(\frac{Y}{\mu_N}-\frac{\mu_Y N}{\mu^2_N}, \frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})}{Var(\frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})} At this point, we have Yc^Nc^=YcNc−θ(XcMc−E[R])\frac{\hat{Y_{c}}}{\hat{N_{c}}}=\frac{Y_{c}}{N_{c}}-\theta( \frac{X_{c}}{M_{c}} - \mathbb{E}[R]) Yt^Nt^=YtNt−θ(XtMt−E[R])\frac{\hat{Y_{t}}}{\hat{N_{t}}}=\frac{Y_{t}}{N_{t}}-\theta( \frac{X_{t}}{M_{t}} - \mathbb{E}[R]) While E[R]\mathbb{E}[R] is hard to deduct, we recognized that the expectation term is the same for both group. We decided to substitute E[R]\mathbb{E}[R] with XcMc\frac{X_{c}}{M_{c}} so the formulas above are transformed to these following two: Ycv(control)Ncv(control)=Y(control)N(control) \frac{Y_{cv}(control)}{N_{cv}(control)}=\frac{Y(control)}{N(control)} Ycv(test)Ncv(test):=Y(control)N(control)−(Y(control)N(control)−θX(control)M(control))+(Y(test)N(test)−θX(test)M(test)):=Y(test)N(test)−θX(test)M(test)+θX(control)M(control) \frac{Y_{cv}(test)}{N_{cv}(test)} \\ :=\frac{Y(control)}{N(control)} - (\frac{Y(control)}{N(control)} - \theta \frac{X(control)}{M(control)}) + (\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)}) \\ :=\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)} + \theta \frac{X(control)}{M(control)} Using the optimal θ\theta, we are hoping to reduce group-level variance by plugging the parameter back in to calculate the adjustment. Please note that across-group θ\theta does not necessarily reduce variance for one group, or the sum of variances of all groups, but in most cases it does. Our simulation shows that 98.3% of metrics saw a decrease by CUPED. Statsig will use CUPED variance when all of the following are met:
  • Core assumptions of the CUPED model are satisfied; this can be violated due to rounding error or other data artifacts
    • E(X_hat) = E(X)
    • The pooled variance of the adjusted population across groups is < the variance of the unadjusted population
  • Enough units have pre-experiment values (> 100)
  • Enough percentage of units have pre-experiment values (> 5%)