One-Sided Tests
How Statsig Warehouse Native uses one-sided hypothesis tests in experiments to detect changes in a pre-specified direction with higher power.
One-sided tests (also known as one-tailed tests and non-inferiority tests)
A one-sided test lets you test for a metric moving in only one direction, which you specify in advance. This trade-off gives you additional sensitivity (or power). It differs from the standard Pulse results, which show two-sided results by default.
Use cases for one-sided testing include detecting regressions in guardrail metrics and testing for a change where only one direction has meaningful business impact. For example, you may be less interested in detecting whether a new feature reduces crash rates, but very interested in learning whether the new feature increases crash rates. In that case, you are willing to forgo detecting the reduction in favor of better detecting the increase.
One-sided tests completely disregard the possibility of detecting the metric moving in the direction that isn't specified, but they give you higher sensitivity in the direction you are looking (which allots all your alpha to testing statistical significance in the one direction of interest). This results in one-sided confidence intervals (CIs) that are narrower in the direction of interest than their two-sided counterparts.
How to enable one-sided tests
When setting up an experiment and identifying metrics to measure, the default setting is to run a two-sided test. To change this, click the metric name on the experiment setup screen. A popup opens where you can modify the test type and indicate the direction you want to measure.
V1 doesn't support Bayesian testing yet.
How to read one-sided test results
Metrics using one-sided tests appear in Pulse very similarly to two-sided tests. The only difference is that Statsig shows a one-sided CI rather than a two-sided CI.
One-sided CIs can be confusing at first. They extend to infinity or negative infinity, which is expected because Statsig only detects changes in the specified direction. As with all CIs, they indicate that the real mean value of the metric likely falls within this range. Because the CI for a one-sided metric is wide, it is equally useful to read the results as indicating high confidence that the mean value doesn't fall outside the CI.
FAQ
Why can't I just run two one-sided tests
Running two one-sided tests would result in a less powerful test. One-sided tests work by allocating the entirety of Type I error (alpha/significance) to one direction. Adding a one-sided test in the other direction reintroduces the chance of making a Type I error in that direction. The result is confidence intervals that appear tighter but actually produce higher rates of decision error than the specified confidence level (default: 95%).
Why use a one-sided test rather than a two-sided test
The choice depends on your use case, metric of interest, and the business impact of any decision. Choose one-sided tests when you plan to act only on changes in one direction and detecting changes in the other direction has no business value.
Was this helpful?