Example: Test GPT4o vs. GPT4o-mini
Step 1: Create configs
Create two dynamic configs, one namedgpt-4o
and another named gpt-4o-mini
. In the Value section add the endpoint, key and other default parameters like this:
Step 2: Create some metrics to track
Let’s take the example of a metric like latency and see how to create it in Statsig. Navigate to the Metrics Catalog page (https://console.statsig.com/metrics/metrics_catalog) and click on Create button.Property | Value |
---|---|
Metric Type: | Aggregation |
ID Type: | User ID |
Aggregation Using: | Events |
Aggregation Type: | Average |
Rollup Mode: | Total Experiment |
Event: | usage |
Average Using: | Metadata => latency_ms |
Step 3: Create an experiment
Create a new experiment in the Statsig console from https://console.statsig.com/experimentsStep 4: Set up the variations
You can now create the control and test variants for the experiment you want to run. In our case, let’s split them evenly 50/50. In the Groups and Parameters section, click on Add Parameter button and name the parameter model_name, with String typeStep 5: Save and start the experiment
Now, hit the Save button at the bottom of the page. You will now see a Start button appear at the top of the experiment page. Go ahead and click it - this will start the allocation process for the experiment.Step 6: Let’s write some code
The code below:- Fetches the experiment configuration from server for a given user. You can pass down the userID from your client application or use one from your database. The code below generates a random one for testing purposes.
- Gets the config name from the experiment variant - either from control or test
- Create a model client using the config that we just fetched
- Uses that model client to complete text.