Skip to main content

Moving from POC to Production

In this guide, we introduce you to transitioning from a proof of concept or limited test on Statsig to a broader production deployment.

1. Check project name/structure​

  1. Rename the project if needed. Most teams continue to use the project they used during trial.

  2. Decide if you need more than one project. Tools like tags and granular permissions let you to manage many teams within a project. Create a new project only for a separate product - when the definition of a user (and DAU) is distinct and no metrics need to be shared. You can require Admins to create new projects so this doesn't happen accidentally. image

  3. Seed your project with a set of tags to use across gates, experiments and metrics.

  4. Create Segments to identify internal employees so you can easily override them into feature rollouts and experiments.

2. Setup SSO to ease onboarding​

Instead of having to invite each user and remembering to deactive them if they leave your company, set up Single Sign On with your identity provider. Statsig supports OIDC - a modern protocol that most identity providers support. Employees (your identity provider verifies) will have a Statsig account created on first use. Once you do this, make your project "Open" so new users verified by SSO default to having access to your projects. If you choose to keep your project closed, you can explicitly approve new requests. image

3. Decide on change management policies for production​

Statsig lets you require reviews before making changes in production. This is a best practice similar to requiring code reviews when developing software. You can also create Review Groups (to notify) and apply more granular permissions to specific feature gates and experiments. Check out our recommended approach to managing non-production environments like dev and staging.

image image

4. Set up Key Metrics and Dimensions​

You should make sure metrics and dimensions you want to experiment on are set up and correct before starting experiments. Some things to consider/explore while determining this include

  • Custom Metrics that build on top of other metrics (e.g. aggregations, ratios)
  • User Dimensions to be able to slice experiment results by Geo, User etc.
  • Metric Dimensions to be able to able slice metrics (eg. add-to-cart events by Product Category)
  • Funnels
  • Tag metrics so that frequently used collections of metrics are easy to bring into experiments

image

Decide if you want to export data from Statsig back into your systems.

Consider if you want to customize Statsig's DAU definition to include only meaningful events. The default is to count a user as active if they generate any event. image

5. Decide what IDs to experiment on​

Most experiments or feature rollouts on Statsig target users (identified by userID). Statsig also allows you choose arbitrary "units of assignment" beyond userID.

  1. userID: Identifies users.
  2. stableID: This is a Statsig client SDK managed ID that is unique to a device. It is typically used to experiment on anonymous users.
  3. You can also create arbitrary units of assignment. Examples include a. organizationID for B2Bs b. VIN or vehicleIDs if you're controlling features on cars c. creatorID if you have a two sides marketplace with creators and consumers (regular users) and want to control by either

Read more

6. Set up integrations​

You might already have set up integrations to bring data into Statsig. Consider other integrations that may be helpful -

  • Export assignment data from Statsig to your data warehouse
  • Export Change History to Slack, Teams or Datadog (overlay over operational dashboards)

7. Best practices, Forums and Training​

Set up a best practice repo and a forum for experimenters to share best practices, get feedback and seek help. Have guidance available to your teams on when to use Feature Gates vs Experiments.

If you've contracted with Statsig for Enterprise Support

  • consider what training specific to your SMEs might be useful and ask for this. Start with train the trainer sessions so your SMEs can drill deeper into topics and then use this to construct guidance relevant to your use
  • Switch support communication to the Slack Support channel with an SLA

Here're some ideas on potential topics to engage on image

8. Determine migration strategy (if any)​

Pick a date after which new experiment and gates are created on Statsig. Allow experiments and ephemeral gates on legacy systems to conclude and wind down there. Move over only gates that are expected to persist for extended periods.