Aggregated Impact
How Statsig Warehouse Native calculates aggregated business impact across metrics for an experiment so you can see the total estimated launch effect.
Metric Insights and Aggregated Impact
Statsig's Insights page provides a clear view of how experiments and feature gates affect a specific metric. It helps answer questions such as "How much impact have I driven?" and serves as a tool for diagnosing unexpected changes in metrics.
Insights presents a reverse perspective of the Pulse view. Pulse measures the impact of a new feature on all your metrics; Insights focuses on a single metric and identifies which tests are affecting it the most. This makes Insights useful for assessing your team's impact and setting realistic goals.How to read Insights
- Navigate to the Insights section in the Statsig console: https://console.statsig.com/. Insights is also available in the insight tab for each metric.
- Select a metric from the drop-down at the top of the page.
- Select the ID type, time window, and other filters.
- Based on the filters you choose, you can see the relative impact, topline impact, and projected launch impact for any experiment or gate that uses this metric.
- Statsig also sums the projected launch impacts, adjusts based on false positive risk ('winner's curse'), and displays the result as the 'Aggregated Impact Estimate'.
How the math works
Refer to the topline impact documentation for details on how topline and projected launch impact are calculated.To estimate false positive risk and calculate Aggregated Impact, Statsig uses the methodology in this paper which is widely adopted across the industry. Specifically:$$
FPR_i = \frac{\alpha_i \times \pi}{\alpha_i \times \pi + (1 - \beta_i) \times (1 - \pi)} $$
In this formula, $ \alpha_i$ is the significance level for experiment i, $ \beta_i$ is the type II error, and 1 - $ \pi$ is the prior success rate based on historical experiment results.
Was this helpful?