Meta-Analysis
The Concept
As teams run a number of experiments, it is possible to glean learning across these experiments. This is meta-analysis. We're rolling out a suite of meta-analysis features this summer - feel free to reach out to help influence our roadmap here.
Experiment Timeline View
This view lets you to filter down to experiments a team has run. At a glance you can answer questions like
- What experiments are running now?
- When are they expected to end?
- What % of experiments ship Control vs Test?
- What is the typical duration?
- Do experiments run for their planned duration - or much longer or shorter?
- Do experiments impact key business metrics - or only shallow or team level metrics?
- How much do they impact key business metrics?
Metric Correlation View
This view lets you visualize two metrics (each data point is an experiment) and visually inspect them for correlation.
Often the metric you want to move isn't very sensitive and takes a while to measure. It is helpful to find metrics that are more sensitive and faster to measure - and run experiments on this.
This view lets you plot two metrics on the same chart - each data point is an experiment's impact on them. You can quickly get a sense for whether the metrics tend to move together - or not. You can also remove outliers, filter down to a team's experiments or download the underlying dataset.
In this hypothetical example - "Checkouts" is the metric you want to move, but it's not very sensitive. "AddToCart" correlates well with "Checkouts", while "ViewItemDetail" doesn't.
Metric Insights
This view lets you pick a metric and see all experiments and feature rollouts that impact this metric. Learn more.