Topline and Projected Impact

How Statsig estimates the topline impact of an experiment on company metrics by scaling experiment lift to your total addressable user base.

Topline impact is the daily change in a company metric that an experiment caused while it ran, measured across the users in the experiment. Projected launch impact scales that number to your whole user base to estimate the daily change if you launch the test group to everyone, relative to the expected baseline value of the metric without the experiment. Use topline impact to report what the experiment did; use projected launch impact to inform a launch decision.

Statsig shows topline impact and projected launch impact in both absolute and relative units. Neither uses CUPED, because CUPED already adjusts for pre-exposure data, which is the same data the topline metric changes from. Applying CUPED would double-count that adjustment.

Take an experiment with a Control group of 1000 users and a Test group of another 1000 users that ran for 30 days. For an event_count metric, the experiment delta is +1.0 events per user (absolute). The topline impact for this metric is +33.33 events per day (absolute).

Computing topline impact

Statsig computes the topline impact over the total duration of the experiment. The calculation depends on whether the metric is an absolute quantity or a ratio.

Topline impact for count and sum metrics

This calculation applies to event_count and sum metrics. Statsig derives the absolute topline impact from the experiment results: the difference in means between test and control, multiplied by the average number of users in the test group per day.

Impact_{abs}=(X_t-X_c) \cdot N_t / n_{days}

Statsig then computes the relative impact from the absolute impact and the overall metric value shown in the metrics dashboard. The relative impact is the percentage change in the overall metric value over the rollup window that Statsig attributes to the active experiment.

Impact_{rel}=\frac{Impact_{abs}}{\text{Topline\_Value}-Impact_{abs}} \times 100\%

Topline impact for ratio and mean metrics

To derive the topline impact on a ratio metric, Statsig first determines the impact on the numerator $X$ and the denominator $Y$ separately. The topline impact is the current value of the ratio metric minus the baseline value obtained by subtracting the numerator and denominator impacts:

Impact_{abs}=\frac{Topline_X}{Topline_Y}-\text{Baseline\_Value}

where the baseline value is the expected value of the topline metric if the experiment weren't running:

\text{Baseline\_Value}=\frac{Topline_X-(\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline_Y-(\bar{Y_t}-\bar{Y_c}) \cdot N_t}

Statsig computes the relative impact for ratio metrics by dividing the absolute impact by the baseline value:

Impact_{rel}=\frac{Impact_{abs}}{\text{Baseline\_Value}} \times 100\%

Computing projected launch impact

Statsig uses the layer allocation of the experiment and the size of the test group to estimate a scaling factor $m$ , which represents the increase in absolute impact expected when you launch the test group. To accommodate changes in allocation during the experiment, Statsig calculates the launch factor over a rollup window as:

m_{rollup}=\frac{1}{\sum_{1}^{rollup}{\text{layer\_alloc} \times \text{group\_pct}}} \times rollup

The calculation doesn't factor in the targeting gate. The projected impact calculation assumes that the targeting gate remains the same after you launch the experiment.

Projected launch impact for count and sum metrics

This calculation applies to event_count, event_dau, and sum metrics. The projected absolute impact is the current topline impact scaled by $m$ . For example, consider an experiment with 50% layer allocation and a 50/50 test/control split, so that 25% of all users are in the test group. If the topline impact is +10 events per day, launching the experiment results in +40 events per day. If the allocation changed during the experiment, Statsig uses a weighted average of the historical allocations.

Projected_{abs}=Impact_{abs} \times m

The relative projected impact is the expected percentage change in the topline metric, relative to the baseline value of the metric without the experiment running.

Projected_{rel}=\frac{Projected_{abs}}{\text{Topline\_Value}-Impact_{abs}} \times 100\% = Impact_{rel} \times m

Projected launch impact for ratio and mean metrics

The projected impact of a ratio metric depends on the numerator and denominator impacts, using the same scaling factor $m$ for each term:

Projected_{abs}=\frac{Topline_X+(m-1) \cdot (\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline_Y+(m-1) \cdot (\bar{Y_t}-\bar{Y_c}) \cdot N_t} - \text{Baseline\_Value}

where the first term is the projected metric value after launch.

The projected relative impact of a ratio metric is the projected absolute impact divided by the baseline value of the ratio:

Projected_{rel}=(\frac{Projected_{abs}}{\text{Baseline\_Value}}) \times 100\%

Confidence intervals for topline and projected impact

Statsig computes the confidence intervals for topline and projected impact with the same method as confidence intervals for experiment deltas.

CI(Impact) = Impact \pm Z \cdot \sqrt{var(Impact)}

For the absolute impact of count and sum metrics, the variance is a linear combination of the test and control variances:

var(Impact_{abs})=[var(\bar{X_t})+var({\bar{X_c}})] \cdot N_t^2

For projected launch impact:

var(Projected_{abs})=var(Impact_{abs}) \cdot m^2

For ratio metrics and relative impacts, Statsig calculates the variance with the Delta method, which accounts for the correlation between the numerator and denominator terms by linearizing non-linear combinations of experiment variables.

For example, the variance in the relative impact of a count metric is:

var(Impact_{rel})=var(Impact_{abs}) \cdot \frac{(\text{Topline\_Value} - 2 \cdot Impact_{abs})^2}{(\text{Topline\_Value} - Impact_{abs})^4}

Was this helpful?