---
title: "p-Value Calculation"
description: "What p-values mean in Statsig experiments, how they are computed, and how to interpret them alongside confidence intervals and lift estimates."
product: general
lang: en
last_updated: 2025-09-18
token_estimate: 926
---
# p-Value Calculation

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

The p-value is the probability of observing a metric delta at least as large as the measured one if the treatment had no effect, which is the null hypothesis. Statsig marks a metric statistically significant when its p-value is below the Type I error threshold $\alpha$ (0.05 by default), using a two-sample z-test, or Welch's t-test when the degrees of freedom $\nu$ are below 100. Use the p-value to decide whether an effect exists; use the [confidence interval](https://docs.statsig.com/experiments/statistical-methods/confidence-intervals) to judge how large it is.

In both tests, the p-value depends on the metric [mean](https://docs.statsig.com/experiments/statistical-methods/metric-deltas) and [variance](https://docs.statsig.com/experiments/statistical-methods/variance) that Statsig computes for the test and control groups.

A p-value below $\alpha$ usually coincides with a confidence interval that doesn't cross zero. For the exception with relative deltas under the Delta Method and Fieller Intervals, refer to [confidence intervals](https://docs.statsig.com/experiments/statistical-methods/confidence-intervals).

## Two-sample tests

### Two-sided z-test

Statsig computes the z-statistic (z-score) of a two-sample z-test. The following forms are equivalent:

$$
\begin{split}
Z &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\overline X_t)+ var(\overline X_c)}} \\
&= \frac{\overline X_t - \overline X_c}{\sqrt{var(\Delta \overline{X})}} \\
&= \frac{\overline X_t - \overline X_c}{\sqrt{\sigma_{\overline{X}_t}^2 + \sigma_{\overline{X}_c}^2}}
\end{split}
$$

where:

- $Z$ is the observed z-statistic, not the z-critical value $Z_{\alpha/2}$.
- $var(\Delta \overline{X})$ is the variance of the absolute delta of means.
- $var(\overline{X}_i)$ is the [variance of the sample mean](https://docs.statsig.com/experiments/statistical-methods/variance) of the control or treatment group.
- $\sigma_{\overline{X}_t}$ is the standard error of the mean of the control or treatment group. Pulse shows these values on the _Statistics_ tab of a metric.

The two-sided p-value comes from the standard normal cumulative distribution function:

$$
p-value = 2 \cdot \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{-|Z|}{e^{-t^2/2}dt}
$$

### Welch's t-test

For smaller sample sizes, Welch's t-test produces lower false positive rates when group sizes and variances are unequal. In Pulse, Statsig applies Welch's t-test automatically when the degrees of freedom $\nu < 100$.

Statsig computes the t-statistic (t-score) with the same formula as the two-sample z-statistic, and the degrees of freedom $\nu$ as:

$$
\nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}
:= \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}
$$

Statsig then obtains the p-value from the t-distribution with $\nu$ degrees of freedom.

### One-sided z-test

For a one-sided z-test, Statsig computes the z-statistic $Z$ with the same formula as the two-sided test, and takes the p-value from the standard normal cumulative distribution function with these differences:

$$
p-value =
\begin{cases} 1 - \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if right-hand test}\\
\frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if left-hand test}
\end{cases}
$$

where $Z$ is the signed z-statistic, not the absolute value that the two-sided p-value uses.