Custom queries can be run on pulse results in order to gain deeper insights from your experiments. These allow you to group by user dimensions, or filter to a specific set of users to see how the experiment impacted their experience.
Be careful with statistical interpretation of custom queries, especially when grouping by a dimension with lots of options. This can increase your chance of seeing a false positive statistically significant results.
Running a Custom Query
To run a custom query, navigate to your pulse results. In a feature gate, custom queries are accessible via the Custom Queries tab, immediately adjacent to the Pulse Results tab. For experiments, you can access custom queries via the Explore tab within Metrics Lifts in Pulse.
Entry point into custom queries for feature gates:
Entry point into custom queries for experiments:
Custom query fields:
- Name: the name of your custom query. This is a display name, so use it to describe your query and why you're running it.
- Date range: the date range for which you're running your analysis on
- User dimension group by: a user dimension that the results will be grouped by. For example, if you were to choose Country, each Country would have its own lift calculated in the custom query
- User dimension fitler: a user dimension that the results will be filtered to. Users not matching this dimension will be excluded from the analysis
- (Advanced) ID List Segment filters: You can choose an ID-list based segment, and your results will only be calculated for users who are in that segment. This can be useful if you forgot to log an important user dimension that you want to filter to, or realized that you only care about a sub-population that you've defined in your own data warehouse
- Careful! This option can easily lead to erroneous and biased results. You will need to make sure the segment is defined based on the user's status before they were exposed to the experiment or feature gate.
Note: These queries take a few minutes to run, but once complete the results will be visible in the "Custom Queries" tab of the Gate or Experiment you ran the query for (you will also receive an email and console notification with a link).
Note: User data in this tool is based off of first-touch attribution. The filters and grouping applied will be based on the user attributes collected at the time of first exposure.
Viewing a Custom Query
These queries take a few minutes to run, but once complete the results will be visible inline within the gate or experiment you ran the query for (you will also receive an email and console notification with a link).