Runs

Runs in Statsig Agent Experimentation grade outputs that already exist, and a run group keeps every run, so you can track scores over time. Each run applies evaluators to the output column of a dataset. The outputs can be answers your agent produced, responses from a model you plan to replace, or production traces. Compare runs to find out whether a change to your agent, model, or evaluators raised or lowered scores.

Use a run on the Runs page when you already have the outputs. To generate outputs from config versions and grade them, start an offline run from the config instead. To grade live traffic as it arrives, use online evaluation.

Runs and offline runs from a config

Both kinds of run grade every row of a dataset. They differ in where the output comes from and in what they compare.

Create a run group

A run group holds a series of runs. Create one run group for each thing you want to track, such as an agent or a regression suite.

1. In the Statsig console, go to Agent Experimentation > Runs.
2. Click Create. If your project has no runs yet, click Create a Run.
3. In the Create New Eval dialog, enter a name and an optional description.
4. Click Create.

Statsig opens the new run group on its Runs tab, with the form for its first run.

The Runs page lists each run group with its name, last modified date, and creator. Search the list, filter it by creator, or sort it by name or last modified date.

To delete a run group, open it, open the more actions menu (⋯) in the header, and click Delete. Statsig deletes the results of every run in the run group.

Create a run

A run grades every row of one dataset with the evaluators you choose. The dataset needs at least one output column, which holds the output to grade.

1. Open the run group's Runs tab. A new run group already shows the run form. Otherwise, click Create above the list of runs.
2. In Run Name, enter a name for the run, such as the change it tests.
3. Under Dataset, click Select a dataset. Pick a dataset from the list, or click Create new local dataset to upload a CSV file and mark its output columns.
4. Click Add Evaluator, and then pick an evaluator from the catalog or click Create new evaluator. Repeat for each evaluator. Evaluators you create here join the Evaluators catalog.
5. (Optional) To change an evaluator's role in the run, open its more actions menu (⋯) and click Set as Primary or Set as Critical Evaluator.
6. Click Create Run.

Statsig grades the rows in the background. The run's card shows a progress bar until grading finishes.

The dataset list shows global datasets and the datasets of earlier runs in this run group. You can't pick a dataset that has no output column. A local dataset that you create in the run form belongs to the run group and doesn't appear on the Datasets page.

Create Run stays disabled until the run has a name, a dataset, and at least one evaluator. It also stays disabled when your company has used up its Included Credits and has no OpenAI API key. For details, go to Evaluator costs.

What evaluators read from each row

Evaluators read the row's output column as {{output}}. If the dataset has more than one output column, {{output}} holds a JSON object that maps each output column name to its value. A column named input fills {{input}}, and a column named trajectory fills {{trajectory}}. Evaluators can also reference any other column by name. For the full list of variables, go to Write a judge prompt.

Primary and critical evaluators in a run

• Primary evaluator: The run's card shows the primary evaluator's score. The first evaluator you add is the primary evaluator.
• Critical evaluator: A must-pass check. When a critical evaluator's score for the run is 0, the run's score is 0, whatever the other evaluators score. For more about critical evaluators, go to What is a critical evaluator.

Review a run

The Runs tab lists the run group's runs on the left. Each run card shows the run's name, its score, and how long ago it started. Search the list with Search runs, and click a run to open it. To delete a run, open the more actions menu (⋯) on its card and click Delete.

A run has two tabs: Run Results, which opens first, and Run Setup.

Results for each evaluator

The Run Results tab shows one card for each evaluator. The card header shows the evaluator's name, type, and score for the run. For evaluators that call a model, the header also shows the average number of tokens the evaluator used for each row. A Primary badge or a Critical chip marks those roles.

Click a card to expand it:

• Score: A gauge with the evaluator's mean score. It shows "No Score" when the evaluator scored no rows.
• Categories: When the dataset's rows have categories, click All or a category tab to filter the score and the rows. A bar chart shows the score for each category. Click a bar to open that category's tab.
• Rows: A table with the dataset's columns, including the output column, and the evaluator's Score for each row. Search the rows, filter them by score, pick the columns to show with Columns, or click Download results as CSV.

A row that an evaluator couldn't score shows N/A. Statsig leaves unscored rows out of the evaluator's score and its category scores.

Click a row to open Row Details. The drawer shows the row's columns and the evaluator's score. For LLM as a Judge evaluators, Judge Response shows the judge's reasoning. Expand Evaluation Prompt to read the evaluator's prompt with the row's values filled in. Press the up and down arrow keys to move between rows.

What a run used

The Run Setup tab shows what the run used:

• Evaluators: A card for each evaluator, with the version that the run used. "Latest version" means the evaluator hasn't changed since the run. Otherwise, the card shows "Version N of M, used for this run" and a View current evaluator link. Expand a card to read the evaluator's definition and targeting rules. Editing an evaluator later doesn't change the run's results.
• Dataset: A preview of the dataset's rows, with search. Icons mark the output and image columns. Click the dataset's name to open its page.
• Parameters: Any parameters that your code sent with its results. Refer to Send results from your code.

Compare runs

The Compare tab shows how scores change across the run group's finished runs. It leaves out runs that are still grading and runs that failed. What the tab shows depends on whether runs share a dataset:

• Runs that share a dataset: Statsig compares them in a card named after the dataset. Choose a Baseline run and one or more Runs to compare. The baseline starts as the earliest run, and the comparison starts as the latest run. For each evaluator and category, Statsig compares the scores of the rows that both runs scored and shows whether the change is statistically significant.
• Runs that don't share a dataset: Each evaluator gets a chart of its mean score over time, after at least two of these runs finish. Each chart has an All tab and a tab for each category. Runs from production traces, and results from your code that don't name a dataset, usually appear here, because each of these runs has its own dataset or none.

Until a second run finishes, the tab shows "A second finished run shows a comparison here."

Grade the past hour of production traces

A project or org admin can turn the past hour of a config's production traces into a run. Statsig grades the run with the config's online evaluators. Use a run from traces to check a config's live outputs on demand.

Start the run in either place:

• From the config: In the config's header actions menu, click Enable Online Eval. In the Enable online eval dialog, click Run online graders for the last hour of traces instead.
• From the config's run group: Open the more actions menu (⋯) in the run group's header and click Run eval from traces. Navigate to original prompt in the same menu opens the config.

Statsig builds the run in four steps:

1. Statsig collects the config's spans from the past hour that have input and output messages. When a trace has a workflow span, Statsig uses only that span. It skips tool spans and the spans that Statsig generates itself.
2. Statsig builds a local dataset with one row per span. The columns are Input, Trajectory, and Output, plus Prompt Version ID and Template Variables. Output is the output column.
3. Statsig grades the dataset with the config's online evaluators.
4. Statsig adds the run to the config's run group. The first time, Statsig creates that run group with the name auto_eval_from_traces_eval: followed by the config's name.

The run's name is the hour it covers, as a timestamp such as 2026-11-04T15:00:00.000Z. Its card shows an Auto chip, and its Run Setup tab explains where the run came from. The scores also appear on the original spans in Traces.

The config needs at least one online evaluator. If no span from the past hour qualifies, Statsig doesn't create a run.

Send results from your code

You can score outputs in your own code and record the results as runs in a run group:

• AI SDKs: The Eval() function in the Node and Python AI SDKs runs your task and scorers on each record, and then sends the results to the run group you name.
• Console API: The Send eval results endpoint, POST /console/v1/evals/send_results/{name}, accepts results that you scored anywhere. Eval() sends its results through this endpoint.

Both methods need an existing run group with the name you send. The Console API calls run groups "evals." Create the run group first, on the Runs page or with Create Eval. If no run group has the name, Statsig rejects the results with a 404 error. Give each run group a unique name, because Statsig finds the run group by name. Requests authenticate with a Console API key.

Each call adds one run to the run group. Statsig names the run with the name you send, or Eval Run N when you don't send one. The results appear on the run as follows:

• Scores: Each named score becomes an evaluator card on Run Results. The first score is the primary evaluator unless the request sets primaryGrader. Scores listed in the request's criticalGraders are critical.
• Categories: Each record's categories become category tabs.
• Summary scores: These appear in a Summary Scores card on Run Results.
• Parameters: These appear on Run Setup.
• Trace IDs: When a result includes a traceID and spanID, its scores also appear on that span in Traces as offline run scores.

To compare runs row by row, name a dataset in the request's dataset field, by its name or ID, and give each result the rowID of a different row of that dataset. Statsig rejects the results when a rowID is missing, repeated, or not a row of the dataset. Runs on the same dataset appear in the dataset's card on the Compare tab. In the Node AI SDK, pass statsigDataset() as Eval()'s data. For details, go to Run an eval on a Statsig dataset. Runs without a dataset appear on the mean score charts on the Compare tab.

To start a run on a dataset with catalog evaluators from code, call Create Eval Run, and then read the results with Get Eval Run Results. The Console API calls configs prompts, so to link a run group to a config, pass the config's name or ID as promptID when you create the run group.

Traces for graded rows

Statsig links each graded row of a run that you create on the Runs page to a trace in Traces. The trace shows the row's input and output next to its scores.

• First run: Statsig creates a small trace for the row in the eval environment.
• Later runs: Statsig adds their scores to the same trace, so the row's scores from every run stay together.
• Edited rows: After you edit a row's values, its next run creates a new trace.
• Rows copied from production traces: A row that Statsig copied from a production trace, and that you haven't edited, adds its scores to the original trace.

These scores appear on the trace's Evaluators tab as offline run scores, with a View runs link to the run. They don't count toward online evaluation results. The Traces page hides the eval environment by default, so remove that filter to find these traces.