Meta-Analysis

How meta-analysis works

As teams run many experiments, patterns emerge across those experiments. Meta-analysis surfaces those cross-experiment insights. Common questions include:

• How hard is a metric to move?
• Are there more sensitive proxies for the metric you care about?
• How are teams doing relative to each other?

Experiment timeline view

This view lets you filter down to experiments a team has run. At a glance you can answer questions like:

1. What experiments are running now?
2. When are they expected to end?
3. What percentage of experiments ship Control vs Test?
4. What is the typical duration?
5. Do experiments run for their planned duration, or much longer or shorter?
6. Do experiments impact key business metrics, or only shallow or team-level metrics?
7. How much do they impact key business metrics?

Metric impact (batting average)

The "batting average" view shows how easy or hard a metric is to move. Filter to a set of shipped experiments and see how many experiments moved a metric by 1% vs 10%. You can filter by team, tag, or statistical significance. Common uses include:

• Validating whether a claim that the next experiment will move this metric by 15% is reasonable.
• Establishing realistic goals based on past ability to move this metric.

Metric correlation view

This view lets you visualize two metrics and inspect them for correlation. Each data point represents one experiment's impact on both metrics.

Often the metric you want to move isn't very sensitive and takes a while to measure. Finding more sensitive, faster-to-measure proxy metrics and running experiments on those proxies can accelerate your work.

You can remove outliers, filter to a team's experiments, or download the underlying dataset.

In this hypothetical example, "Checkouts" is the metric you want to move, but it isn't very sensitive. "AddToCart" correlates well with "Checkouts", while "ViewItemDetail" doesn't.

Metric insights

Knowledge Bank

The Knowledge Bank is a searchable repository of experiment learnings across teams. It helps you find shipped, healthy experiments, gain context on past work, and generate ideas for new experiments.

New team members can explore experiments a team ran or search by topic. The meta-analysis tools offer more structured ways to discover and review your experiment corpus. The Knowledge Bank is available when you need free-text search.