Datasets

Create and edit datasets for Statsig Agent Experimentation evals: upload a CSV, mark output and image columns, categorize rows, and add production traces as test cases.

A dataset is a table of test cases that Statsig Agent Experimentation evals grade. Each row is one test case, such as a user question and the answer you expect. Each column is a variable that prompts and graders reference by name, such as {{input}}. Use a dataset to grade an AI Config version, or outputs your agent already produced, against the same cases every time. To grade live production traffic instead, use online evals.

A dataset holds test cases, not graders. An AI Config's version evals use the graders attached to the AI Config. A run on the Evals page uses the graders you pick when you create the run. Refer to Choose where a grader runs.

Global and local datasets

Agent Experimentation has two kinds of dataset:

Kind	Where you create it	Listed on the Datasets page
Global	The Datasets page, AI Traces, or Make Dataset Global in an AI Config	Yes
Local	An AI Config's Eval Setup, or a new local dataset for a run on the Evals page	No

Any AI Config or eval can use a global dataset. A local dataset belongs to the AI Config or run that created it.

Create a dataset from a CSV file

Go to Agent Experimentation > Datasets and select Create.
Enter a Dataset Name and an optional Description. Each global dataset in a project needs a unique name.
Drag a CSV file onto the upload area, or select Upload CSV and choose the file.
Check the preview, then select Upload.

Statsig creates a global dataset and opens its page. To mark output or image columns, use the column menu after the upload. Refer to Output and image columns.

CSV format

The first row of the file holds the column names, and each later row becomes one test case:

csv

input,expected_output,categories
"My order arrived damaged. What should I do?","Apologize and offer a replacement.","shipping,refunds"
How do I reset my password?,Send the password reset link.,account

Statsig converts column names to lowercase, replaces spaces with underscores, and removes every other character that isn't a letter, digit, or underscore. For example, a column named Expected Answer becomes expected_answer, which you reference as {{expected_answer}}.

Two column names have special behavior:

Column	What Statsig does with it
`categories`	Reads a comma-separated list of categories for each row and stores them as row categories, not as a regular column. Optional.
`annotations`	Suggests categories from the annotation text when more than five rows have a value. Optional. Refer to Suggested categories.

The upload accepts one CSV file of up to 2 MB and 1,000 rows. Statsig skips empty rows.

Add a dataset to an AI Config

Every version of an AI Config runs its offline evals against the same dataset. To choose that dataset, open the AI Config's Setup tab. In the Eval Setup group, under Dataset, select Select a Dataset, then choose one option:

Upload Dataset: Upload a CSV file as the AI Config's local dataset.
Browse from Existing Datasets: Search the global datasets and pick one.
AI Dataset Generation: Describe the test cases you want in a prompt and set Total rows, from the default of 20 up to 300. Select Preview Dataset to inspect the rows, or Apply Dataset to use them as the AI Config's local dataset.
Make Dataset Global: Convert the AI Config's local dataset into a global dataset. The dataset then appears on the Datasets page, and other AI Configs and evals can use it.

The dataset table under Dataset works like the table on a dataset's page, and you save its edits with the AI Config. Each kind of change affects the dataset differently:

Upload or generate a dataset: Statsig replaces the rows of the AI Config's local dataset. If the AI Config uses a global dataset, Statsig detaches it and creates a new local dataset. The global dataset doesn't change.
Switch to another dataset: Statsig deletes the rows of the AI Config's local dataset. To keep those rows, select Make Dataset Global before you switch.
Edit a global dataset: Edits you make from inside an AI Config change that dataset for every AI Config and eval that uses it.

Each upload, generated dataset, or edit starts a new dataset version for the AI Config. The AI Config's Evals tab compares two runs only when both used the same dataset version.

Edit rows and columns

Open a dataset from the Datasets page to edit it on the Setup tab. Your edits apply only after you select Save in the save bar, and Discard Changes reverts them. If you try to leave the page with unsaved changes, Statsig asks you to confirm.

To work with rows:

Filter the table with Search.
Select Add Row, fill in each column in the drawer, and confirm the row.
Select a row to open it in a drawer. Select Edit on a column, change the value, and select Save. Press the up and down arrow keys to move between rows.
Select a row's delete icon to remove it. A dataset must keep at least one row.

To work with columns, open the menu in a column header:

Add Column: Add a new column.
Mark as Output or Unmark as Output: Set or clear the column's output flag.
Mark as Image or Unmark as Image: Set or clear the column's image flag.
Delete Column: Remove the column. A dataset must keep at least one column.

To rename a column, select its name and type a new one. Each column needs a unique name that isn't empty.

Output and image columns

An icon in the column header shows each flag:

Output column: Holds an output your agent already produced. A run on the Evals page grades the values in the output column. A run can use a dataset only after you mark at least one output column. Evals that you start from an AI Config generate their own output, so they don't need an output column.
Image column: Holds image URLs. Statsig sends those images to LLM-as-a-judge graders along with the grading prompt.

When you create a local dataset for a run on the Evals page, the upload drawer also shows Output Columns and Image Columns. That drawer requires at least one output column.

Categorize rows

Categories group rows so a run on the Evals page can show scores for each category. A row's categories appear in the categories column at the end of the table.

Add a category to a row: Hover over the row's categories cell, select the add icon, and pick a category. To create a category, type a name of up to 30 characters and select the Create option.
Remove a category from a row: Select the delete icon on the category chip.
Rename or delete categories: Select Manage Categories at the bottom of the category picker. Edit the list, select Add New Category to add one, and then select Save.

Suggested categories

Statsig suggests categories when you create a dataset in which more than five rows have an annotations value. An LLM reads each annotation and proposes short categories for its row. Datasets you create from annotated traces often qualify. The console calls these suggestions tags.

After the suggestions are ready, the toolbar shows two options:

Apply all suggested tags: Add every suggested category to its rows.
Review Results: Open Suggested Tags, which lists each suggestion with the annotations behind it. Clear the suggestions you don't want, and then apply the rest.

When any row has an annotations value, All and Annotated tabs appear above the table. The Annotated tab shows only the rows that have annotations.

Add traces from AI Traces

AI Traces can turn production spans into test cases:

Go to Agent Experimentation > AI Traces.
Select one or more spans and select Add to Dataset. You can also open a trace and select Add to dataset.
Pick a global dataset, or find one with Search datasets. To start a new dataset, select Create new Dataset, enter a Dataset Name, and select Create.

Each selected span becomes one row with these columns:

Column	Value
`input`	The span's input messages from `gen_ai.input.messages`, one `role: text` line per message.
`output`	The text of the final assistant message in `gen_ai.output.messages`, without tool calls or earlier steps.
`metadata`	JSON with the model, token counts, AI Config name (`promptName`), trace and span IDs, status, and timing.
`annotations`	JSON list of the span's annotations. Statsig adds this column only when the span has annotations.

The span's tags become the row's categories. When you add spans to an existing dataset, Statsig adds any of these columns that the dataset lacks. The dataset keeps its output and image flags. A new dataset from AI Traces has no output column. To use it in a run on the Evals page, select Mark as Output on the output column.

View the runs that used a dataset

A dataset's Results tab lists the runs on the Evals page that used the dataset, with columns for Score, Run, Eval, and Started. A run that's still grading shows a Running badge, and a failed run shows an Error badge. Evals that you start from an AI Config don't appear here; their results are on the AI Config's Evals tab. Refer to Offline evals.

Export or delete a dataset

Open the more-actions menu in the dataset's header:

Export to CSV: Download the rows as a CSV file. When rows have categories, the file includes a categories column, so you can upload it again. The file doesn't record output or image flags.
Delete: Delete the dataset after you confirm.

Was this helpful?