On this page

Pulse

Pulse is Statsig's visualization of how an experiment impacted target metrics. This page walks through the tool and how to use it to evaluate your experiment.

You can navigate to your Pulse Results from within any experiment or gate. This is also the default landing page for an experiment that has an analysis ready to view.

Exposures

At the top of pulse is the exposures view. This shows how unique users have onboarded to an experiment over time. It also has the overall splits of users in the experiment.

In the health checks, an SRM check flags if exposures are imbalanced in a significant way.

Exposures Card

Scorecard

The scorecard is the core view of pulse.

For a detailed look at the stats engine, go to the stats engine docs.

For every metric, Statsig shows you:

  • The calculated relative difference (Delta %)
  • The confidence interval
  • Whether the result is statistically significant
    • Positive lifts are green
    • Negative lifts are red
    • Non-significant results are grey

The formula for calculating lift is:

Delta(%) = (Test - Control) / Control

Confidence intervals are reported at the selected significance level (95% by default). In a typical two-sided Z-test, Statsig shows the confidence interval as +/- 1.96 * standard error.

Statsig applies Winsorization based on metric-level settings. Winsorization caps extreme outlier values to reduce their impact on experiment results. Statsig applies CUPED (regression adjustment) by default; toggle it off to view non-CUPED results. Sequential testing, Bonferroni corrections for multiple comparisons, family-wise error rate control, and Bayesian analysis are available on the settings page.

Scorecard

Health Checks

The Diagnostics tab shows the results of statistical and data quality checks. These include:

  • Job status. You can also view and copy the SQL that ran for every step of the Pulse analysis.
  • Exposure balance between groups. A chi-squared test detects meaningful differences between group populations and the expected split.
  • Overlap check: Statsig discards users with exposures in both test and control from Pulse analysis. A large overlap population can reduce power or cause bias.
  • Confirmation that exposures were present in the selected analysis range.
  • Confirmation that metric data was joined to exposure data. A failure here can indicate a mismatch in IDs or a misconfigured metric.

Health Checks

Explore

The Explore tab lets you use user dimensions to drill down and explore results. This is a useful tool for diagnostics and hypothesis generation. Treat these results as inconclusive unless the experiment had a specific hypothesis for the target population. You can also select from "special" tags including All Metrics (all metrics in your project) and Scorecard Metrics (all metrics on the experiment scorecard).

Pulse explore tab interface

Was this helpful?