Metric Insights and Aggregated Impact
How Statsig calculates the aggregated business impact of an experiment across metrics so you can quantify total launch impact in one summary view.
Metric Insights and Aggregated Impact
Statsig's Insights page provides a clear view of how experiments and feature gates impact a specific metric. It helps answer key questions such as "How much impact have I driven?" and helps diagnose unexpected changes in metrics.
Insights presents a reverse perspective of the Pulse view. While Pulse measures the impact of a new feature on all your metrics, Insights lets you focus on a single metric and identify which tests are impacting it the most. This is useful for assessing your team's impact and setting realistic goals.How to read Insights
- Navigate to the Insights section on the Statsig console: https://console.statsig.com/ . Insights is also available in the insight tab for each metric.
- Select a metric from the selector drop-down at the top of the page.
- Select the ID type, time window, and other filters.
- The page shows the relative impact, topline impact, and projected launch impact for any experiment or gate that includes this metric.
- Statsig sums the projected launch impacts, adjusts based on false positive risk ('winner's curse'), and displays the result as the Aggregated Impact Estimate.
How the math works
Go to the topline and projected launch impact calculations for details.To estimate false positive risk and calculate Aggregated Impact, Statsig uses the methodology in this paper, which is widely adopted across the industry. Specifically:$$
FPR_i = \frac{\alpha_i \times \pi}{\alpha_i \times \pi + (1 - \beta_i) \times (1 - \pi)} $$
In this formula, $ \alpha_i$ is the significance level for experiment i, $ \beta_i$ is the type II error, and 1 - $ \pi$ is the prior success rate based on historical experiment results.
Was this helpful?