Running a Warehouse Native POC

What Statsig Warehouse Native does

Statsig Warehouse Native enables customers with existing metric logs to run analysis on their existing metric data, and optionally bring previous assignment data or offline experiments into the platform. Statsig WHN has two types of experiments:

1. Assign and Analyze: You can run an experiment on web/mobile/app and use Statsig’s SDKs to assign (bucket or randomize) users, and then analyze results.
2. Analyze: You can run an experiment elsewhere (your own SDK, email, direct mail, sms, ivr, etc.) and use Statsig to analyze that data and calculate experiment results. Statsig reads this assignment data from your warehouse in this format; Statsig calls these Assignment Sources.

Steps to running an effective proof of concept with Warehouse Native

Follow these steps when planning your Warehouse Native implementation:

1. Define your experiment(s) and metrics for validation - A proof of concept determines if Statsig fits your experimentation needs; running an experiment is the most direct path for evaluation. Responsible Party: Typically a product or engineering lead.

   • Plan to run 1-2 production level experiments to validate. Past experiments, A/A tests or upcoming projects or product changes are great opportunities to implement a Statsig experiment!
     • Identify your hypothesis and metrics to validate this hypothesis. Statsig joins these metrics with unit assignment/exposure data and runs them through the stats engine.
     • If your team plans on running analysis only, identify the user assignment data that Statsig joins with the metric data.
       • This approach can yield results for analysis in as little as 30 minutes, assuming data is readily available for ingestion
     • If your team plans on using the Assign and Analyze experimentation option, identify where the experiment will run. Typically, web-based experiments are easier to evaluate, but Statsig also has SDK support for server and mobile SDKs.
       • Note: It’s important the implementing team understands how the SDKs operate before executing a proof of concept. The client and server docs can help orient your team.
       • A typical evaluation takes 2-4 weeks to account for experiment design, implementation, time to bake, and analysis. To ensure a successful POC, have a well scoped plan and ensure the right teams are included to assist along the way.
   • Read experimentation best practices to get an idea of how to best succeed.

2. Connect the warehouse - To query data and operate within your warehouse, you’ll need to allocate resources and connect to Statsig. You may choose to utilize an existing prod database or create a separate cluster specifically for experimentation (if you don’t already have one).

   • Statsig requires a role and the following access:
     • Read access to metric and exposure data
     • Write access so Statsig can write results and exposures back to the warehouse
     • Access to run jobs and query data
       • Find more guidance on connecting with your specific warehouse vendor here.
   • Review the data pipeline overview to understand how data flows for warehouse native jobs.

3. Connect metric sources and define metrics - Once the data warehouse has been connected, you can begin defining metric and assignment sources (if applicable) in Statsig. Statsig expects specific schemas to correctly map the data to its pipelines:

   

   Beyond these columns, the schema is flexible and can accept additional columns. You can use Metadata to filter metrics and perform more granular analysis.

   • Review guides for creating a metric and assignment source
   • Follow the data best practices to ensure your queries are running efficiently.

   This section is important to review and can prevent unnecessary infrastructure costs!

   After metric sources have been connected, configure metrics to perform various aggregations (e.g., Sum, Mean, Count, Unique Users) that represent what you’re trying to measure in your experiments.
   • Supported metric types and ways to configure them
   • Use cohort metrics to measure impact during a certain time frame per user.
4. Create and roll out an experiment - In step 1, you defined the planned experiment(s) and the metrics used to validate them. With metrics and assignment sources configured, you can now create the experiment(s). Refer to the detailed guide for experiment setup, and consider these things as you complete this step:
   • Create your hypothesis and select the experiment (assignment) source
     • If using the SDK for assignment, the SDK itself serves as the assignment source.
   • Custom IDs can be used but must first be configured; for example: device_id, vehicleId.
   • Refer to advanced settings for the many ways you can configure your experiment
   • As you roll out your experiment, you can monitor the status with health checks and get a readout of live exposures as they come through the SDK.
   • If you’re hoping to quickly validate the platform, you can create and run a quick A/A test.
5. Read results - Once the experiment has successfully run, read the results and ensure everything looks reasonable. Was your hypothesis validated or are the results surprising? Are the results easy to interpret and navigate for the teams involved? Refer to the section on Pulse to get an idea of the high-level analytics capabilities. A few things to note:
   • You can slice results further using the Explore tab, which lets you break down results by specific user and event properties.
   • You can configure exposure and metric data to be forwarded to your warehouse.
   • The Health Checks (diagnostics) tab surfaces the SQL used to generate results so you can validate any analysis performed on your systems.

6. Finalize evaluation and next steps: Ultimately, a POC validates a set of evaluation criteria that determines whether Statsig is a good fit for your team’s workflows. The following graphic provides high-level guidance on what to look for during your evaluation phase.