Be careful when drawing your inferences of Custom Queries, especially when grouping by a dimension with lots of options. This can increase your chance of seeing a false-positive statistically significant result.
Dimension Loading Timing for Precomputed User Dimensions
When viewing results for precomputed user dimensions (which are configured and run on a schedule), be aware that these dimensions are loaded through separate asynchronous explore queries. This means:- The main experiment results will appear first
- Precomputed dimensions will continue loading in the background and become available within a few minutes
- This timing gap is most noticeable immediately after the first reload of the day
- If you see “No dimensions available for this time range” for precomputed dimensions, wait a few minutes and refresh to see if dimensions have completed loading
This timing behavior only affects precomputed user dimensions that run on a schedule. User-triggered custom queries do not experience this asynchronous loading delay.
Running a Custom Query
To run a Custom Query, navigate to the Explore tab within your experiment.- Metric(s): The metric(s) you want to analyze. You can select a single metric, a few metrics, or a Metric Tag. Adding a Tag will include all the metrics within that Tag in your Custom Query. There are three “default” metric selections included as shortcuts:
- “Scorecard Metrics”, all metrics included in your experiment setup’s Primary and Secondary Metrics sections
- “Primary Metrics”
- “Secondary Metrics”
- Metric Filter: With metrics selected, you can filter metrics by either Event or User dimensions using the “Add Filter” dropdown. For example, if you wanted to look at your experiment results for Canadian users only, you could filter to “Country = CA”.
- Group By: You can group your Custom Query results by either an Event or User dimension. Whereas Custom Query filters can be applied at the per-metric level, the Group By action is at the query level (so all included metrics will have whatever Group By you select applied to them).
- Time Range for Metric Data: The date range you’re running your analysis on. By default this will be the “Full date range” of your experiment data.
- (Advanced) ID List Segment filters: You can choose an ID-list based Segment, and your results will only be calculated for users who are in that segment. This can be useful if you forgot to log an important user dimension that you want to filter to, or realized that you only care about a sub-population that you’ve defined in your own data warehouse.
- Careful! This option can easily lead to erroneous and biased results. You will need to make sure the segment is defined based on the user’s status before they were exposed to the experiment or feature gate.
- Similarly, you can choose to exclude a certain ID list segment, for example if you want to exclude a set of users who have been retroactively identified as bad actors from your lifts analysis.
- (Advanced) Filter by Exposure Date: You can also filter the results by Exposure Date which can give you more flexibility. You can choose only include or exclude a date range, or in WHN, you can additionally include/exclude users based on when they were first exposed to the experiment.
- This is useful when your metrics have novelty effect, delayed impact, or specific scenarios where you only want to filter your results to certain users. Use it cautiously because it can lead to biased results.
User groups in experiment results are based off of first-touch attribution. The filters and grouping applied will be based on the user attributes collected at the time of first exposure in the gate/experiment/layer check.