One-Sided Test
How Statsig uses one-sided hypothesis tests in experiments to detect changes in a pre-specified direction with higher statistical power.
One-sided tests (aka one-tailed test, non-inferiority test)
A one-sided test lets you test for a metric moving in only one direction, which you specify in advance. This trade-off gives you additional sensitivity (or power) compared to the standard Pulse results, which show two-sided results by default.
Use cases for one-sided testing include detecting regressions in guardrail metrics and testing for a change where only one direction has meaningful business impact. For example, you may not care whether a new feature reduces crash rates, but you do care whether the new feature increases crash rates. In that case, you are willing to forgo detecting the reduction in favor of better detecting the increase.
One-sided tests don't detect metric movement in the unspecified direction. They give higher sensitivity in the direction of interest by allocating all alpha to that direction. This results in one-sided confidence intervals (CIs) that are narrower in the direction of interest than their two-sided counterparts.
How to enable one-sided tests
When setting up an experiment and identifying metrics to measure, the default setting is to run a two-sided test. To change this, click the metric name on the experiment setup screen. A popup opens where you can modify the test type and indicate the direction you want to measure.
V1 doesn't support Bayesian testing yet.
How to read one-sided test results
Metrics using one-sided tests appear in Pulse similarly to two-sided tests. The difference is that Pulse shows a one-sided CI rather than a two-sided CI.
One-sided CIs extend to infinity or negative infinity, because Statsig only detects changes in the specified direction. As with all CIs, the one-sided CI indicates that the true mean value of the metric likely falls within that range. Because the CI is wide on the unbounded side, you can also interpret the result as high confidence that the mean doesn't fall outside the CI.
FAQ
Why can't I run two one-sided tests?
Running two one-sided tests results in a less powerful test. One-sided tests work by allocating all Type I error (alpha) to one direction. Adding a second one-sided test in the other direction reintroduces the chance of a Type I error in that direction. This produces confidence intervals that appear tighter than warranted, leading to higher decision error rates than the specified confidence level (default 95%).
Why use a one-sided test rather than a two-sided test?
The choice depends on your use case, the metric of interest, and the business impact of any decision. Select a one-sided test when detecting a change in only one direction is valuable and you can accept not detecting changes in the other direction.
Was this helpful?