Best Practices and Avoiding False Positives

Sound experiment readouts start from a written hypothesis, a small set of Scorecard metrics, and skepticism toward borderline p-values. Follow these practices when you read a Scorecard. For why a result can flip from significant to neutral as data accumulates, refer to the Pulse FAQ.

Start with a hypothesis

Decide before you open the Results tab which metrics you expect the change to move, what else could have happened, and what would signal that something went wrong.

Pick a small set of key metrics

Choose a few metrics directly tied to your hypothesis that establish whether the experiment worked. More than a handful of key metrics usually indicates an ill-defined hypothesis or unfocused experimentation.

Avoid cherry-picking results

Don't select three metrics that look good while ignoring two that don't, and don't pick "good" or "bad" numbers with no connection to your hypothesis. A statistically significant result should have a plausible explanation, and a false positive is one plausible explanation.

Look for consistent supporting effects

Multiple independent effects that fit one plausible story lend credibility to the observed effects, even when individual p-values are borderline.

Expect false positives

Examining many metrics increases the false positive rate. At a 95% confidence interval (5% significance level), expect about one statistically significant metric in twenty from random chance alone, and more if you include borderline metrics (for example, p = 0.06). Be suspicious of statistically significant results with borderline p-values.

Look beyond your hypothesis

Check for other effects, tradeoffs, and unexpected behaviors. These findings show how users interact with your product and often lead to follow-up experiments and new ideas.