# p-Value Calculation

In Null Hypothesis Significance Tests, the p-value is the probability of observing an effect at least as large as the measured metric delta. The p-value assumes the null hypothesis is true. In practice, a p-value below the pre-defined Type I Error threshold ($\alpha$) serves as evidence of a true effect.

The methodology for p-value calculation depends on the number of degrees of freedom ($\nu$). A two-sample z-test is appropriate for most experiments. Statsig uses Welch's t-test for smaller experiments with $\nu \< 100$. In both cases, the p-value depends on the metric [mean](https://docs.statsig.com/experiments/statistical-methods/metric-deltas) and [variance](https://docs.statsig.com/experiments/statistical-methods/variance) computed for the test and control groups.

Typically, a p-value below threshold $\alpha$ occurs only when the confidence interval doesn't cross 0. However, this correspondence isn't always the case in the Statsig UI. When the p-value of the difference between test and control is significant, the relative delta confidence interval may still cross zero (when using [The Delta Method](https://docs.statsig.com/experiments/statistical-methods/methodologies/delta-method)) or appear as a point estimate (when using [Fieller Intervals](https://docs.statsig.com/experiments/statistical-methods/methodologies/fieller-intervals)). In these cases, the absolute difference's p-value remains statistically significant.

## Two-sample tests

### Two-sided z-test

You can compute the z-statistic (also called the z-score) of a two-sample z-test in multiple equivalent formats:

$$
\begin\{split}
Z &= \frac\{\overline X\_t - \overline X\_c}\{\sqrt\{var(\overline X\_t)+ var(\overline X\_c)}} \\\\
&= \frac\{\overline X\_t - \overline X\_c}\{\sqrt\{var(\Delta \overline\{X})}} \\\\
&= \frac\{\overline X\_t - \overline X\_c}\{\sqrt\{\sigma\_\{\overline\{X}\_t}^2 + \sigma\_\{\overline\{X}\_c}^2}}
\end\{split}
$$

where:

* $Z$ is the observed z-statistic (not the z-critical value $Z\_\{\alpha/s}$)
* $var(\Delta \overline\{X})$ is the variance of the absolute delta of means
* $var(\overline\{X}\_i)$ is the variance of sample means either control or treatment group (details [here](https://docs.statsig.com/experiments/statistical-methods/variance))
* $\sigma\_\{\overline\{X}\_t}$ is the standard error of the mean of either control or treatment group (these are the terms you can find in Pulse under the Statistics tab of a metric)

Statsig obtains the two-sided p-value from the standard normal cumulative distribution function:

$$
p-value = 2 \cdot \frac\{1}\{\sqrt\{2\pi}} \int \limits \_\{-\infty}^\{-|Z|}\{e^\{-t^2/2}dt}
$$

### Welch's t-test

For smaller sample sizes, Statsig prefers Welch's t-test because it produces lower false positive rates when group sizes and variances are unequal. In Pulse, Statsig automatically applies Welch's t-test when the degrees of freedom $\nu \< 100$.

Statsig computes the t-statistic (also called the t-score) identically to the two-sample z-statistic above. Statsig computes the degrees of freedom $\nu$ using:

$$
\nu = \frac\{\left(var(\overline X\_t) + var(\overline X\_c)\right)^2}\{\frac\{var(\overline X\_t)^2}\{N\_t - 1}+\frac\{var(\overline X\_c)^2}\{N\_c - 1}}\
:= \frac\{var(\Delta\overline\{X})^2}\{\frac\{var(\overline X\_t)^2}\{N\_t - 1}+\frac\{var(\overline X\_c)^2}\{N\_c - 1}}
$$

Statsig then obtains the p-value from the t-distribution with $\nu$ degrees of freedom.

### One-sided z-test

The one-sided z-test computes the z-statistic $Z$ in the same way as the two-sided test. The one-sided p-value differs as follows:

$$
p-value =
\begin\{cases} 1 - \frac\{1}\{\sqrt\{2\pi}} \int \limits \_\{-\infty}^\{Z}\{e^\{-t^2/2}dt} &\text\{if right-hand test}\\\\
\frac\{1}\{\sqrt\{2\pi}} \int \limits \_\{-\infty}^\{Z}\{e^\{-t^2/2}dt} &\text\{if left-hand test}
\end\{cases}
$$

where:

* Statsig computes $Z$ as in the two-sided test. This uses the signed z-statistic, not the absolute value used in the two-sided p-value.