Metric Alerts

When rolling out a new feature or experiment, you may want to be notified if your rollout or experiment regresses a metric beyond a pre-set threshold. Statsig enables this via metric alerts.

Overview

Metric alerts are hourly for the first 24 hours post experiment/ gate rollout (this includes new gate rules/ rollouts), and then daily thereafter. This is to help provide more real-time visibility during the most critical phase of a rollout; in those first 24 hours post-going live. Alerts only trigger if the metric delta is statistically significant lower/ higher than your threshold, which helps reduce alert noisiness.

Finally, all stats methodologies you've enabled for your experiment/ gate rollout (CUPED, Sequential Testing, etc.) will be applied to alert calculations post-the first 24 hours (once the alert becomes daily).

NOTE: Metric alerts do not alert at the topline metric value level, but rather at the experiment/ feature gate level. This means that even if you have an experiment allocated to 10% of your users, but the metric change within that 10% allocation breaches the set threshold, you will be alerted. All alerts you receive will be in the context of a specific experiment or feature gate and to debug/ resolve the alert you will be directed to the offending experiment or gate in question.

Setting up a Metric Alert

To set up a metric alert, go to the Metrics tab —> Metrics Catalog and search for the desired metric.

Once in the Metric Detail View, go to the Alerts tab, and tap + Create Alert. As part of Metric Alert configuration, you will be asked to configure the following inputs-

• % Change- The metric delta threshold after which you want to trigger an automated alert. As noted above, this delta is in the context of the feature gate rollout or experiment the metric is being measured in and is not a top-line metric value change across your whole user-base.
• Minimum Participating Units- You can set a minimum threshold of exposures you want to log before triggering an alert. We surface the 25/ 50/ 75th percentiles of exposure volume across your gates/ experiments to help you choose a reasonable threshold.
• Direction- Positive or negative, depending on whether you want to be alerted based on your metric exceeding vs. dropping below a target threshold.
• Subscribers- By default, all creators of a feature gate/ experiment with a metric that has an alert configured will be notified if an alert is fired. You can also add additional, global subscribers to a metric alert, who will be notified if any feature gate or experiment regresses the metric beyond the target threshold. Note that in a large project adding yourself as a global subscriber of a metric risks being noisy.

Once a metric alert has been configured for a given metric, you will see an “alert” alarm bell icon next to the metric inline in the Metrics Catalog. You can also filter for metrics with alerts set on them via the standard metrics filtering affordance next to the search bar.

Determining the Right Threshold

To help you configure the right threshold for your metric alert, there is a preview of how much the metric has moved in the context of any feature gates or experiments containing that metric in the Scorecard.

To see how a given metric has trended over a longer period of time, hover over the metric delta for a given feature gate/ experiment, and tap on a data point to view more details. This will open the time-series for the metric, with a configurable date range picker.

Alert UX

If a Metric Alert is triggered, all subscribers and the relevant gate/ experiment creator(s) will receive a notification via email, in the Statsig Console, and via Slack for users who have configured Slack notifications for their Statsig accounts.

Tapping on View Alert will take you into the Diagnostics page of the offending feature gate or experiment that triggered the metric alert.

Once in the Diagnostics section, scroll to the Metric Delta Alerts section and select the active alert. You will see the metric delta trend with the threshold overlayed. To resolve the alert, tap Resolve inline, which will give you the option to either resolve the alert (+ provide an explanation), or snooze the alert for a specified period of time. The alert will show as resolved both inline in the Metric Delta Alerts section, as well as on the top of the Pulse page for the given gate/ experiment.

Viewing Alert History

To view alert history, go to Metrics tab —> Metrics Catalog and select the metric you want to view alert history for. Then go to Alerts —> … menu in the Experiment and Gate Alerts section, and select View Alert History. You will be able to see all instances of the alert firing, being resolved, or being snoozed, as well as reason provided for resolution, who resolved the alert, and at what time.