One-Sample Test

One-sample tests (fixed-value test)

A one-sample test compares a single sample of data against a known or hypothesized value to determine whether there is a statistically significant difference. Unlike A/B tests that compare two groups, one-sample tests evaluate whether a single group differs from a specific benchmark, target, or historical baseline.

When to use one-sample tests

One-sample tests are useful for comparing a single group against a known value:

• Single Group Events: When only one group can trigger certain events (for example, feature usage or error types), compare against an expected baseline.
• Algorithm Testing: Test whether an algorithm performs better than random (for example, whether a success rate differs from 50%).

Statistical considerations

One-sample tests provide a way to make statistical inferences about whether observed data differs significantly from a hypothesized value. The test helps determine whether any observed difference is due to random variation or represents a true change in the underlying process.

Enable fixed-value baseline comparison

1. Go to the setup page of an experiment

2. Click the metric name

3. Select Use Fixed Baseline as Control