Interaction Detection

What is interaction detection

When you run overlapping experiments, the experiments can interfere with each other. Interaction Detection lets you pick two experiments and evaluate them for interaction. This helps you understand whether users exposed to both experiments behave differently from users exposed to only one. The core question is: Does being in both experiments change the effect on a metric in a way that differs from summing the individual effects?

Should I worry about it?

How to use interaction detection

To start an Interaction Detection analysis, go to Experiment > Results > Explore > Interaction Effect Detection.

Select a second experiment and the metrics to analyze. Statsig then shows a summary with three sections:

• By Groups: unit counts based on the collective slice of group assignment
• Metric Summary Results: estimated intervals of difference in metric lift
• Overlapping Unique Users: an overview of the general traffic intersection of the two experiments

Methodology

Assume you have two experiments A and B, each with two groups: control and test. The interaction effect calculates the overlapping impact on users exposed in both experiment A and experiment B.

Intuition

• The part $\overline{X_{testA \cdot testB}} - \overline{X_{testA \cdot controlB}} $ answers: “If someone is in A’s test, how much does changing B from control to test change the metric?”
• The part $\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}}$ answers: “If someone is in A’s control, how much does changing B from control to test change the metric?”
• Subtracting those two values shows whether the effect of B differs depending on whether the user is in A’s test or control group. That difference is the interaction effect.

If the result isn’t statistically significant, the effect of B is the same regardless of which group the user is in for A (no significant interaction). If it is statistically significant, the effect of B depends on A’s assignment (significant interaction).

Directionality

The direction (positive or negative) of the interaction effect indicates whether the combination of the two treatments amplifies or dampens each other’s effects.

The interaction effect is defined as:

$Interaction Effect=(Effect of B when A=Test)−(Effect of B when A=Control)$

• Statistically positive interaction: B’s effect is stronger when A is also active.
• Statistically negative interaction: B’s effect is weaker (or reversed) when A is active.

Magnitude

The magnitude (absolute size) of the interaction effect indicates how much the combined impact deviates from simple additivity.

For example:

• Experiment A produces a +5% lift on your metric of interest
• Experiment B produces a +5% lift on your metric
• The interaction effect between A and B is –3%

This means that when both experiments are rolled out together, their overall impact is expected to be about 3% lower than the sum of their individual effects: the features interfere with each other.

To obtain the most accurate estimate of the true combined impact, run a new experiment that includes both features together. Because experiments A and B may not have run over exactly the same time period or under identical conditions, differences in timing and seasonality can slightly influence the measured interaction magnitude.