Note: this solution is still functional, but can be manual and time consuming to set up with minimal error handling. We encourage you to check out the Data Warehouse Ingestion solution instead.
There are 2 ways to integrate with Redshift: using a data connector, or ingesting events and metrics to Statsig through S3.
Using a Data Connector
To ingest events from Redshift, you can use our Census integration.
To export events to Redshift, you can use our Fivetran integration.
S3 imports are currently a custom setup flow. You'll need to reach out to Statsig through Slack or through your support PoC in order to set up this integration. The documentation below describes the steps to set up this integration. There are 3 main steps:
Create a pipeline to write your metric, event, and (optionally) signal data to an S3 bucket in parquet format
Create an IAM user with read and list access on that bucket and send that user's Key/Secret to Statsig. We will securely store these in a keystore service
Schedule ingestion through a
signalsdataset or through the
Set up a data pipeline to S3
We will expect data in your S3 bucket to be saved in parquet format.
To allow for daily uploads, please set up your bucket with the following folders:
events/for events data
metrics/for metrics data
signals/for signal flags when you've finished uploading data for a day. You can omit this folder and instead use the
mark_data_readyAPI instead, but you must use one or the other
We recommend writing folders by date partitions for ease of debugging, i.e. storing day's data in folders with ISO-formatted names (
Please make sure your data conforms to the following schemas.Events
|timestamp||UNIX timestamp of the event||UTC timestamp|
|event_name||The name of the event||String under 128 characters, using |
|event_value||A string representing the value of a current event. Can represent a 'dimension' or a 'value'||Read as string format; numeric values will be converted into value|
|event_metadata||A dictionary<string, string> in the form of a JSON string, containing named metadata for the event||String format|
|user||A JSON object representing the user this event was logged for; see below||Escaped JSON string including the keys 'custom' and 'customIDs'. A userID or customID must be provided.|
|timeuuid||A unique UUID or timeUUID used for deduping. If omitted, will be generated but will not be effective for deduping||UUID format|
Please refer to docs for the Statsig User Object for available fields. An example would look like:
userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.40 Safari/537.36",
Make sure to include all of metric_value, numerator, and denominator, writing
cast(null as double) for numerator and denominator if you are omitting them (or conversely for metric_value if sending numerator/denominator).
|unit_id||The unique user identifier this metric is for. This might not necessarily be a user_id - it could be a custom_id of some kind||String format. Make sure this is in the same format as your logged unit_ids|
|id_type||The id_type the unit_id represents.||String format. Must be a valid id_type. The default Statsig types are user_id/stable_id, but you may have generated custom id_types. Make sure this matches (case sensitive) a customID in your project, or you won't get experiment results|
|date||Date of the daily metric||Read as string format; can be written as ISO date. Statsig's dates are calculated in PST - we'll load custom metrics to whatever date you use here|
|metric_name||The name of the metric||String format. Not null. Length < 128 characters|
|metric_value||A numeric value for the metric||Double format. Metric value, or both of numerator/denominator need to be provided for Statsig to process the metric. See details below|
|numerator||Numerator for metric calculation||Double format. Required for ratio metrics. If present along with a denominator in any record, the metric will be treated as ratio and only calculated for users with non-null denominators|
|denominator||Denominator for metric calculation||Double format. See above|
Set up and Provide Credentials
- Navigate to your IAM console on AWS
- Go to Users->Add User
- Select the
Access key - Programmatic accesscredential type
- Attach an appropriate policy which gives Read and List access to the appropriate bucket. Make sure this is scoped appropriately so the user only has access to the data intended! Example policy:
Next, modify your bucket access policy (under
permissions on your S3 bucket's page) to allows this user to access objects.
You can confirm your credentials are sufficient by adding any data to your metrics folder and running the following code in PySpark with the IAM user credentials:
Because you may be streaming events to your tables or have multiple ETLs pointing to your metrics table, Statsig relies on you signalling that your metric/events for a given day are done.
To do this, write a dataset with the single column
finished_date, which contains all dates of data which have been written to Statsig. For example, once you have written data for
2022-06-22 you would insert a record with
2022-06-22 to trigger ingestion of data from up to and including
Unlike some other integrations like Snowflake, for S3 Statsig will skip dates; if your latest finished date is
2022-06-22 and you insert
2022-07-01, we will ingest all data as of
2022-07-01 and infer that data for dates between (e.g.
2022-06-25) is loaded.
Alternatively, you can use the
mark_data_ready API and send a timestamp for which the data previous to that timestamp has finished loading into S3.
Note that, for events, Statsig processes days according to PST. When you mark data ready for '2022-06-20', statsig will process events from
2022-06-20T00:00 PST to
2022-06-20T23:59.... PST. Keep this in mind when scheduling your signals!