Playground

The Statsig Playground runs up to 10 prompts on the same inputs and shows their outputs side by side. Each output includes the latency and token count of the response. Use the Playground to try different models, messages, and parameters before you create an AI Config. When a prompt works the way you want, promote the playground's prompts to a new AI Config.

The Playground doesn't grade outputs, and it runs on 10 inputs at most. To score AI Config versions against a full dataset with graders, run an offline eval instead.

Create a playground

1. In the Statsig console, go to Agent Experimentation > Playground. The list shows each playground's name, last modified date, and creator.
2. Click Create.
3. In the Create New Playground dialog, enter a name. You can also add a description, a team, and tags.
4. Click Create.

Statsig opens the new playground on the Setup tab. A new playground has one blank prompt and an empty inputs table with one column, input.

To delete a playground, open the more actions menu (⋯) in the header and click Delete. Statsig deletes the playground with its prompts and inputs. You can't undo a deletion.

Set up prompts

The Setup tab shows the playground's prompts side by side. Statsig labels them A, B, C, and so on, in order. Each prompt has its own provider, model, parameters, and messages.

To configure a prompt:

1. Select a provider and a model from the menus next to the prompt's letter.
2. To change model parameters, click the options icon next to the model. You can set Temperature, Top P, Frequency Penalty, Presence Penalty, and Max Tokens.
3. Write the prompt's messages. Set the role of each message to System, User, or Assistant. To add a message, click Message.
4. To use an input in a message, click its variable in the Variables list below the messages. For example, {{input}} refers to the input column. When the playground runs, Statsig replaces each variable with the row's value.

To add a prompt, click Clone Prompt at the top of the tab. Statsig adds a copy of the last prompt. To copy or remove a specific prompt, open that prompt's more actions menu (⋯) and click Clone Prompt or Delete Prompt. A playground holds 1 to 10 prompts.

Start from an AI Config

To start from a prompt you already serve, click Start from config on a prompt on the playground's Setup tab, then select an AI Config. Statsig copies the provider, model, parameters, and messages of the AI Config's Live version into the prompt. The menu shows the Live version number of each AI Config. The menu doesn't list AI Configs whose Live version uses a custom workflow.

The copy isn't linked to the AI Config. Edits to the prompt don't change the AI Config, and later changes to the AI Config don't update the prompt. To reset the prompt to a blank prompt, select None in the same menu.

Add inputs

A playground's Results tab holds the inputs table. Each column is a variable that prompts can reference. Each row is one input, and every prompt runs on every row.

• To add a row, click Add New Input. The table holds up to 10 rows.
• To edit a value, click its cell and type.
• To rename a column, click its name. Column names must be unique.
• To add, clone, or delete a column, open the column's more actions menu (⋯).
• To remove a row, click its delete icon.

Load inputs from a dataset

To fill the table from a dataset, click Select a Dataset and choose one of these options:

• Browse from Existing Datasets: Statsig copies the first 10 rows of the dataset you select. If the dataset has more than 10 rows, a message shows how many of its rows Statsig copied.
• Upload Dataset: Statsig keeps the first 10 rows of the CSV file you upload.
• AI Dataset Generation: Statsig generates rows with AI.

Each option replaces the playground's columns and rows. The playground keeps its own copy of the rows, so edits in the playground don't change the original dataset.

Save and run a playground

Statsig keeps unsaved changes when you switch between a playground's Setup and Results tabs. If you leave the page with unsaved changes, Statsig asks you to confirm. The header has these controls:

• Save Changes (save icon): Saves the prompts and inputs. You can save a playground that has empty messages or empty rows. To save, column names must be unique and not empty, and parameters must be in the ranges listed in Playground limits.
• Discard Changes (reset icon): Returns the playground to its last saved state.
• Start: Saves any unsaved changes, runs every prompt on every row, and opens the Results tab.

Start stays disabled until the playground can run. Hold the pointer over Start to show the reason. A playground can run when all of these conditions are true:

• Every prompt message has content.
• Every prompt that uses Google or Anthropic has at least one User message.
• The inputs table has at least one row, and every row has a value in at least one column.
• Included Credits remain, or you've added an API key for each provider that the prompts use. For details, go to Credits and API keys for playground runs.

Each run replaces the previous results. You can't save the playground while a run is in progress.

Read run results

After a playground run, the Results tab shows one output column for each prompt, labeled with the prompt's letter. Each output cell shows the response, the latency in seconds, and the token count.

Click an output cell to open the Run Details drawer for that row. For each prompt, the drawer shows:

• Metrics: The latency and token count.
• Inputs: The row's value for each column.
• Output: The full model response.
• Resolved Prompts: The messages Statsig sent to the model, with each variable set to the row's value.

After a run, editing an input value or changing the columns clears the affected outputs. Those cells show "Click start to generate output" until the next run. Changes to prompts don't clear existing outputs, so click Start again to get outputs for a changed prompt.

Credits and API keys for playground runs

Each playground run calls the model provider of every prompt. If you haven't added your own API key for a prompt's provider, the run uses your Included Credits. When Included Credits run out, Start stays disabled until you add an API key for each provider that the prompts use. An admin adds keys in Settings > Product Configuration > General, under AI Provider API Keys. For how Included Credits work, go to Credits and API keys.

Promote prompts to an AI Config

To turn a playground's prompts into an AI Config:

1. In the header, click Promote to Config.
2. In the Create New AI Config dialog, review the name. The name defaults to the playground's name. You can also add a description, a team, tags, and target applications.
3. Click Create.

Statsig creates the AI Config and opens it. Prompt A becomes the Live version. Every other prompt becomes a Draft version, in the same order. Promote to Config uses the prompts as they appear on the Setup tab, including unsaved changes. The playground doesn't change.

Save results as a dataset

To keep the outputs of a playground run, open the more actions menu (⋯) in the header and click Save results. Statsig creates a global dataset and opens it. The dataset has the playground's input columns, plus one output column for each prompt, named A, B, and so on. The dataset name starts with saved_playground_results:, followed by the playground ID and a timestamp.

Statsig disables Save results before the first run and while a run is in progress.

Review playground history

To review changes to a playground, click the history icon in the header. The Playground History dialog lists each change, such as saves and runs, with the person who made it and the time. To open a read-only view of a change, click View Changes on its entry.

Playground limits