Creating Custom Metrics
Custom metrics are computed by Statsig from your raw events. To create custom metrics, navigate to Metrics from the left-hand navigation panel, then to the Metrics Catalog tab. Tap on the Create button.
Statsig supports five types of custom metrics:
| Metric Type | Description | Examples |
|---|---|---|
| Event Count | Total count of events filtered by the value and metadata properties of an event type | Add to Cart event filtered by category type |
| User Count | Number of unique users that trigger events filtered by the value and metadata of an event type | Active Users based on their views of a product category |
| Aggregation | Sum or Average of the value or metadata property of an event type | Total Revenue |
| Ratio | Rates (e.g. cart conversion rate, purchase rate), Normalized Values (e.g. sessions per user, items per cart) | Cart Conversion Rate, Sessions per User |
| Funnel | Funnels- funnel of multiple events with conversion tracking | Sign-up Funnel, Checkout Funnel |
Statsig computes custom metrics on a per day basis for your Metrics dashboard, and rolled up for the duration of the experiment in your Pulse Results delivered with your Feature Gates and Experiments. After you create a custom metric, it will not populate until the next day (and will not backfill to previous days). Statsig will only calculate it moving forward from the creation date.
Statsig offers a feature allowing users to set a daily max cap for specific metrics. With this capability, users can define maximum caps for various unit types associated with a metric. Any value surpassing the set cap will automatically be adjusted to match it. For instance, if you determine that purchases greater than $10,000 per day on your E-commerce platform should not skew analysis, any transaction exceeding this threshold will be adjusted downward to this limit, ensuring the integrity of your experiment analysis. Capped metrics are available for event count and aggregation (sum) metric types.
When creating your custom metric, you can preview a given custom metric's value at the bottom of the page. To view metric preview, tap View Output Preview to see what your metric would have looked like based on its components’ values over the last 7 days, in both chart and table form.
Examples
1. Event Count Metrics
Here's an example of setting up a custom event metric to count the number of add_to_cart events filtered by a metadata property called value, which carries the price of the item added to the cart. As this example specifies the ID Type as userID, Statsig will compute this metric as part of the test group that the corresponding user is assigned to in an experiment.
If you select the ID Type as stableID, Statsig will compute this metric in the test group that the corresponding device is assigned to in an experiment. When you select more than ID Type, Statsig will compute this metric for each type of ID Type that you specify.
2. User Count Metrics
The example below creates a metric to count the number of unique users who viewed a product in the toys category that was priced under $10.
Time Window
Two time window options are available for user count metrics
- Daily Participation Rate: It counts the total number of days that a user has the selected event, divided by the number of days the user is in the experiment. The result is a metric value between 0 and 1, which represents the probably of a user having the event on a daily basis. It works best for events that are expected to occur repeatedly for a given user.
- One Time Event: This checks if a user has the selected event at least once during their time in the experiment. The result is a binary metric with value 0 or 1 for each user. This is ideal for events that are only expected once per user, such as sign up events.
- Custom Attribution Window: Allows you to define a custom window after exposure to count an event towards a metric calculation.
3. Aggregation Metrics
The example below shows a Total Revenue metric that sums the value associated with all purchase events.
[In Beta] Currency Sum
eCommerce customers can span multiple currency types and typically log payments in the currency and amounts the user actually pays. Currency sums allow for payments that are recorded in different currencies to be converted to one common currency. You will need to log two pieces of metadata in your event - a currency code and a value. Example metadata - _{"currency_code": "USD", "currency_value": 123.45}_
Statsig supports currency sums upon request. Please ask in Slack.
4. Ratio Metrics
The example below shows the creation of a Cart Conversion Rate metric. Here we use the unique users who triggered the purchase event as the numerator and the unique users who triggered the add to cart event in the denominator. Note that when calculating the numerator, we filter to only include users who also had the denominator event in the same day. So in the case of this metric, a user who only has purchase event on a given day without an add to cart on that same day will not count towards the numerator.
This pattern also applies to click through rates (CTR) in any part of a step-wise product journey (aka funnels). Statsig recommends using unique users in both the numerator and denominator for defining these kinds of metrics. As an example, when a user reloads a page multiple times but clicks only once, this corresponds to a 100% CTR (1 out of 1). Similarly, a user who loads a page once but clicks multiple times on a button should only count as 1 out of 1. This also solves for cases where users see an important button such as "Sign-up" multiple times a day, and we would still consider it a success if they click just once.
The example below shows creating a metric for Items per Cart. You can track the number of unique items added to a cart if you log an add_to_cart event for each item. For the numerator, select total event count. For the denominator, select unique users. As this metric is computed daily and only for users with a non-zero denominator, this metric can generate ratios such as 1/1, 2/1, 2/1, and 5/1 for individual users. When aggregated, this translates to 10/4 = 2.5 items per cart on average per day.
In experimentation, ratio metrics are a frequent source of misleading information. It's possible to see an increase in click through rate alongside a net decrease in total clicks (the opposite may also happen). This situation can occur if the number of unique users viewing a button (denominator) decreases. As a best practice, Statsig recommends tracking the numerator and denominator as independent metrics when monitoring ratio indicator. Ratio metrics are often subject to statistical noise and can be tricky to use for obtaining a statistically significant result. In addition, for the numerator in ratios, we exclude units which don't have a denominator value.
5. Funnel Metrics
You can create a custom funnel metric, from either the Custom Metrics Creation wizard in the Metrics Catalog or via the Charts tab.
Components of Funnel Metrics
Funnel metrics have a few components:
- Lineage: Surfaces the events used to generate the funnel
- Metric Value: Metric value represents the overall funnel conversion rate, or the percentage of users who complete a funnel (trigger the end event) relative to all users who start the funnel (trigger the starting event)
- Conversion rate between stages: This set of metrics track the percentage of users who triggered an event N relative to all users that triggered event N-1 in the funnel
After funnels are created and populated, you can view your funnel metric much like any other metric in Pulse. Additionally, you can expand the funnel metric to view Pulse performance at each step in the funnel.
In the example below, the Square variant shows a lift in the overall funnel conversion rate. Expanding the metrics to examine the entire funnel reveals two key insights:
- Both the Square and Circle variants show a lift in top-of-funnel DAU (Land Page View Start DAU). However, only the Square variant shows statistically significant increase in end-of-funnel DAU (Purchase Event End DAU).
- The overall funnel conversion rate improvement for Square is primarily due to the higher conversion from Checkout Event to Purchase Event stages in the funnel.
Custom Metrics & Dimensions
For Custom Metrics that are composed of a single event (event count/ event dau, aggregation, etc.), the dimensions of the source event will be automatically pulled through into the Custom Metric and be exposed as dimensions of that Custom Metric. However, if your Custom Metric is composed of 2+ different events, any associated dimension(s) of the source input events will not be pulled through as dimensions of the Custom Metric and are also not queryable today via the "Explore" tab in Pulse Results.