One-Sample Test

A one-sample test compares one experiment group's metric against a fixed baseline instead of against a control group, and reports whether the difference is statistically significant. In Statsig, you set the fixed baseline for a metric with the Use Fixed Baseline as Control option on the experiment setup page. The scorecard then compares the group against that value. Use a one-sample test when only one group can produce the metric, or when you're testing against a known target such as a 50% success rate. Use a standard two-group test whenever a real control group exists.

When to use a one-sample test

A one-sample test fits these situations:

• Single-group events: When only one group can trigger certain events (for example, feature usage or error types), compare that group against an expected fixed baseline.
• Algorithm testing: Test whether an algorithm performs better than random (for example, whether a success rate differs from 50%).

Enable a fixed baseline for a metric

1. Go to the setup page of an experiment.
2. Click the metric name.
3. Select Use Fixed Baseline as Control.

The following screenshots show the metrics section of the experiment setup page, the metric name menu, and the fixed baseline dialog.