For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.
One-Sided Test
How Statsig uses one-sided hypothesis tests in experiments to detect changes in a pre-specified direction with higher statistical power.
A one-sided test (also called a one-tailed test or non-inferiority test) checks whether a metric moved in one direction that you specify in advance. It allocates the entire significance level () to that direction. This gives the test more power to detect a change in that direction than the two-sided test Pulse runs by default. It also produces one-sided confidence intervals (CIs) that are narrower in the direction of interest. The trade-off is that a one-sided test never detects movement in the other direction. Use a one-sided test when detecting a change in only one direction is valuable and you can accept missing changes in the other direction, for example regressions in guardrail metrics. Keep the default two-sided test when either direction matters for your decision.
For example, you may not care whether a new feature reduces crash rates, but you do care whether it increases them. In that case, you forgo detecting the reduction in favor of better detecting the increase.
Enable a one-sided test
By default, Statsig runs a two-sided test for each metric you add to an experiment. To change the test type, click the metric name on the experiment setup screen. In the popup, change the test type and select the direction you want to measure.
V1 doesn't support Bayesian testing yet.
Read one-sided test results
Metrics using one-sided tests appear in Pulse similarly to two-sided tests, except that Pulse shows a one-sided CI rather than a two-sided CI.
One-sided CIs extend to positive or negative infinity, because Statsig only detects changes in the specified direction. As with all CIs, the one-sided CI indicates that the true mean value of the metric likely falls within that range.
Why running two one-sided tests reduces power
Running two one-sided tests results in a less powerful test. A one-sided test allocates all Type I error () to one direction. Adding a second one-sided test in the other direction reintroduces the chance of a Type I error in that direction. This produces confidence intervals that appear tighter than warranted, and decision error rates higher than the specified confidence level (default 95%) implies.
Was this helpful?