Topline and Projected Impact

How Statsig Warehouse Native estimates the topline impact of an experiment on company metrics by scaling experiment lift to your total user base.

The topline impact is the average daily effect that an experiment has on the overall metric value, measured between two groups. The projected launch impact is an estimate of the daily global metric impact expected if you launch the test group to all users (beyond those in the experiment). Statsig computes this impact relative to the expected baseline metric value without the experiment running.

Statsig shows both Topline Impact and Projected Launch Impact in absolute and relative units. Neither uses CUPED, because CUPED already adjusts for pre-exposure data (the starting point for topline metrics), and combining the two double-counts that adjustment.

Example: Take a simple example experiment with a Control group of 1000 users and a Test group of another 1000 users, which ran for 30 days. For an event_count metric, the observed Experiment Delta is +1.0 events per user (abs). The Topline Impact for this metric is +33.33 events per day (abs).

Computing topline impact

Statsig computes the topline impact over the total duration of the experiment, which produces the most accurate estimate and tight confidence interval. The calculation depends on whether the metric represents an absolute quantity or a ratio:

Count and sum metrics (event_count, sum)

Statsig derives the absolute topline impact from the experiment results. It depends on the difference in means between test and control and the average number of users in the test group per day.

Impact_{abs}=(X_t-X_c) \cdot N_t / n_{days}

Using the absolute impact and the overall metric value (as shown in the metrics dashboard), Statsig computes the relative impact: the percentage change in the overall metric value over the rollup window attributed to the active experiment.

Impact_{rel}=\frac{Impact_{abs}}{\text{Topline\_Value}-Impact_{abs}} \times 100\%

Ratio and mean metrics

To derive the topline impact on a ratio metric, Statsig first determines the impact on the numerator (X) and denominator (Y) separately. The topline impact is the current ratio metric value minus the baseline value obtained by subtracting the numerator and denominator impacts:

Impact_{abs}=\frac{Topline_X}{Topline_Y}-\text{Baseline\_Value}

Where the baseline value is the expected value of the topline metric if the experiment wasn't running:

\text{Baseline\_Value}=\frac{Topline_X-(\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline_Y-(\bar{Y_t}-\bar{Y_c}) \cdot N_t}

Statsig obtains the relative impact for ratio metrics by dividing the absolute impact by the baseline value:

Impact_{rel}=\frac{Impact_{abs}}{\text{Baseline\_Value}} \times 100\%

Computing projected launch impact

Statsig uses the layer allocation of the experiment and the size of the test group to estimate a scaling factor m, which represents the increase in absolute impact expected when you launch the test group.

Statsig calculates the launch factor over a rollup window as

m_{rollup}=\frac{1}{\sum_{1}^{rollup}{\text{layer\_alloc} \times \text{group\_pct}}} \times rollup

to accommodate changes in allocation during the experiment.

Statsig doesn't factor in the targeting gate. The projected impact calculation assumes that the target gate remains the same after you launch the experiment.

Count and sum metrics (event_count, event_dau, sum)

For count and sum metrics, the projected absolute impact is the current topline impact scaled by factor m. For example, consider an experiment running at 50% layer allocation with a 50/50 test/control split, so that 25% of all users are in the test group. If the allocation changed during the experiment, Statsig uses a weighted average based on historical allocations. If the topline impact is +10 events per day, launching the experiment produces +40 events per day.

Projected_{abs}=Impact_{abs} \times m

The relative projected impact is the expected percentage change in the topline metric, relative to the baseline metric value without the experiment running.

Projected_{rel}=\frac{Projected_{abs}}{\text{Topline\_Value}-Impact_{abs}} \times 100\% = Impact_{rel} \times m

Ratio and mean metrics

The projected impact of ratio metrics depends on the numerator and denominator impacts in the same way as the topline impact calculation. Statsig uses the same scaling factor m to obtain the projected impact for each term:

Projected_{abs}=\frac{Topline_X+(m-1) \cdot (\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline_Y+(m-1) \cdot (\bar{Y_t}-\bar{Y_c}) \cdot N_t} - \text{Baseline\_Value}

Where the first term represents the projected metric value after launch.

Finally, the projected relative impact of a ratio metric is the projected absolute impact divided by the baseline value of the ratio:

Projected_{rel}=(\frac{Projected_{abs}}{\text{Baseline\_Value}}) \times 100\%

Confidence intervals

Statsig computes the confidence intervals for topline and projected impact using the same method as the confidence intervals for experiment deltas.

CI(Impact) = Impact \pm Z \cdot \sqrt{var(Impact)}

For the absolute impact of count and sum metrics, the variance is a linear combination of the test and control variances:

var(Impact_{abs})=[var(\bar{X_t})+var({\bar{X_c}})] \cdot N_t^2

And for projected launch impact:

var(Projected_{abs})=var(Impact_{abs}) \cdot m^2

For ratio metrics and relative impacts, Statsig calculates the variance using the Delta method. This accounts for the correlation between numerator and denominator terms by using Taylor expansion to linearize expressions containing non-linear combinations of experiment variables.

For example, the variance in the relative impact of a count metric is:

var(Impact_{rel})=var(Impact_{abs}) \cdot \frac{(\text{Topline\_Value} - 2 \cdot Impact_{abs})^2}{(\text{Topline\_Value} - Impact_{abs})^4}

Was this helpful?