Get Started with Geotests

This guide walks you through creating a Geotest experiment using Statsig. The steps include:

1. Define your Geo Types
2. Configure your Metric Source
3. Create and Configure
4. Evaluate Design Options
5. Run Analysis

Define your geo types

First decide on one or more geo types relevant to your business. Some of the most common are Postal Codes and DMAs (Designated Market Areas), but Statsig allows you to define any arbitrary geo type.

Configure your metric source

Create and configure

• Navigate to your Experiments page.
• Click the + Create button.
• In the modal, select Analyze to analyze existing experiment data in your warehouse.

• Enter a name and description for your experiment.
• Choose Geotest as the experiment type from the dropdown. Fill in the remaining fields as usual.

In the Setup tab:

• Define your hypothesis.
• Set the Geo Type (e.g., country_id) from one of the types you've defined in your project and metric sources
• Choose your Primary Metrics.

Configure:

• Expected Treatment Start/End Dates
• Pre-treatment duration (days)
• Effect Size % (the MDE you want to measure; smaller MDEs will make finding a valid design harder)
• Type of Test: 1-sided or 2-sided
• Alpha level (the type I error rate you want, inverse of significance)
• Optionally enable Budgeting
• Additional advanced modeling parameters for users familiar with Geolift's Advanced API

Generate design options

When your initial settings are ready, click Use Experiment Designer to begin the design process.

The Experiment Designer automates the process of evaluating potential splits of geos. Creating a new design set involves 3 steps:

1. Define your targeting
2. Set any Inclusion or Exclusion overrides
3. Define your date range

Define targeting

Targeting allows you to define the set of geos eligible to be included in your experiment. By default, Statsig pulls the unique set of geo types from your metric source from the last 90 days. You can add custom filters based on columns in your metric source. For example, you can filter geos to those in a certain region or by transactions that occurred in a certain language.

You can also manually specify a set of geos yourself. The geo id types must match those in your metric source exactly.

Set inclusion or exclusion overrides

After defining your targeting set, you can manually assign any of these geos to treatment or control groups.

For example, when testing a new marketing campaign, you might want to include Austin because a contract has already been signed, and exclude New York, Los Angeles, or Washington D.C. because of local regulations. Specify these constraints in the designer and all design options considered will conform to these rules.

Define date range

Define the data range of your design dataset. This date range defines what data is pulled as the historical baseline to train a synthetic control model. A good guideline is that your design data duration should be at least 4x the expected duration of your treatment period, to achieve good fits and maximize sensitivity and power of your experiments.

Advanced options

• Rolling Lookback Windows: Control how many iterative simulations should be done to evaluate power. For instance, a value equal to 5 would simulate power for the last five possible tests in the date range. Corresponds to lookback_window in the GeoLift API.

• Dynamic Time Warping: Control how much the synthetic control model depends on auto-correlations in the metric vs. correlations between different geos on the same days. A value of 1 focuses exclusively on the metric while a value of 0 (default) relies on correlations only. Corresponds to dtw in the GeoLift API.

• Control Cell Size Requirements: Sets limits on the share of primary metric value from geos in any considered control group. If not set, Statsig analyzes all design options regardless of their size. For example, if designing on a Revenue metric, a control size range of [50%, 90%] means that only design options where 50% to 90% of revenue is in the designated control geos. Corresponds to holdout in the GeoLift API.

Evaluate design options

Once a Design Option Set is created (usually takes a few minutes), Statsig shows it in your Experiment Designs tab. All design option sets Statsig generates appear here, so you can look through prior design iterations as needed.

Click into a completed design set (e.g. “Design Set 3”).

• Review the ranked recommendations for your best design options
• Compare MDE, Power, Cost and Control/Test Allocation
• Click View Cell Details to expand and see specific geo assignments
• Review additional model performance details
• Select your design using the radio button, then click Save Design to Experiment.

Run analysis