On this page

For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.

p-Value Calculation

What p-values mean in Statsig experiments, how they are computed, and how to interpret them alongside confidence intervals and lift estimates.

The p-value is the probability of observing a metric delta at least as large as the measured one if the treatment had no effect, which is the null hypothesis. Statsig marks a metric statistically significant when its p-value is below the Type I error threshold α\alpha (0.05 by default), using a two-sample z-test, or Welch's t-test when the degrees of freedom ν\nu are below 100. Use the p-value to decide whether an effect exists; use the confidence interval to judge how large it is.

In both tests, the p-value depends on the metric mean and variance that Statsig computes for the test and control groups.

A p-value below α\alpha usually coincides with a confidence interval that doesn't cross zero. For the exception with relative deltas under the Delta Method and Fieller Intervals, refer to confidence intervals.

Two-sample tests

Two-sided z-test

Statsig computes the z-statistic (z-score) of a two-sample z-test. The following forms are equivalent:

Z=X‾t−X‾cvar(X‾t)+var(X‾c)=X‾t−X‾cvar(ΔX‾)=X‾t−X‾cσX‾t2+σX‾c2\begin{split} Z &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\overline X_t)+ var(\overline X_c)}} \\ &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\Delta \overline{X})}} \\ &= \frac{\overline X_t - \overline X_c}{\sqrt{\sigma_{\overline{X}_t}^2 + \sigma_{\overline{X}_c}^2}} \end{split}

where:

  • ZZ is the observed z-statistic, not the z-critical value Zα/2Z_{\alpha/2}.
  • var(ΔX‾)var(\Delta \overline{X}) is the variance of the absolute delta of means.
  • var(X‾i)var(\overline{X}_i) is the variance of the sample mean of the control or treatment group.
  • σX‾t\sigma_{\overline{X}_t} is the standard error of the mean of the control or treatment group. Pulse shows these values on the Statistics tab of a metric.

The two-sided p-value comes from the standard normal cumulative distribution function:

p−value=2⋅12π∫−∞−∣Z∣e−t2/2dtp-value = 2 \cdot \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{-|Z|}{e^{-t^2/2}dt}

Welch's t-test

For smaller sample sizes, Welch's t-test produces lower false positive rates when group sizes and variances are unequal. In Pulse, Statsig applies Welch's t-test automatically when the degrees of freedom ν<100\nu < 100.

Statsig computes the t-statistic (t-score) with the same formula as the two-sample z-statistic, and the degrees of freedom ν\nu as:

ν=(var(X‾t)+var(X‾c))2var(X‾t)2Nt−1+var(X‾c)2Nc−1:=var(ΔX‾)2var(X‾t)2Nt−1+var(X‾c)2Nc−1\nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} := \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}

Statsig then obtains the p-value from the t-distribution with ν\nu degrees of freedom.

One-sided z-test

For a one-sided z-test, Statsig computes the z-statistic ZZ with the same formula as the two-sided test, and takes the p-value from the standard normal cumulative distribution function with these differences:

p−value={1−12π∫−∞Ze−t2/2dtif right-hand test12π∫−∞Ze−t2/2dtif left-hand testp-value = \begin{cases} 1 - \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if right-hand test}\\ \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if left-hand test} \end{cases}

where ZZ is the signed z-statistic, not the absolute value that the two-sided p-value uses.

Was this helpful?