---
title: Evaluators
description: "Create and configure evaluators in Statsig Agent Experimentation to score AI output in offline runs and online evaluation, with LLM judges, Python, string, and similarity checks."
product: general
lang: en
token_estimate: 4544
---
# Evaluators

An evaluator scores AI output with a number from 0 to 1. Statsig runs evaluators in two places: offline runs, which grade every row of a dataset, and online evaluation, which grades a sample of live traffic. Evaluators live in one catalog for your company. Any [config](https://docs.statsig.com/agent-experimentation/configs) or run can reuse an evaluator, so its scores stay comparable across versions.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Use an evaluator when you want Statsig to compute the score. To record a score that your own code computes, such as user feedback, call `logEvalGrade` from the [Node AI SDK](https://docs.statsig.com/agent-experimentation/node#logging-eval-results) instead.

The Console API still calls evaluators "graders." For example, you create an evaluator with [Create Eval Grader](https://docs.statsig.com/api-reference/evals/create-eval-grader).

## Evaluator types

Every evaluator has one of four types. The type decides how the evaluator scores output and whether it can run in online evaluation.

| Type | How it scores | Runs online |
| --- | --- | --- |
| LLM as a Judge | A model reads your prompt and the output, then returns a score from 0 to 1 with its reasoning. | Yes, when the prompt uses only `{{input}}`, `{{output}}`, and `{{trajectory}}` |
| String Comparison | Compares `{{output}}` with a dataset column using Equals, Does not equal, Contains, or Contains (ignore case). Scores 1 when the comparison holds and 0 when it doesn't. | No |
| Text Similarity | Scores how similar `{{output}}` is to a dataset column, using Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, or Cosine. Cosine embeds both texts with the OpenAI `text-embedding-3-large` model. | No |
| Python | Runs your `grade(sample, item)` function, which returns a number. | Yes, when the code reads only `item["input"]` and `item["trajectory"]` |

A row or span passes an evaluator when it scores 0.67 or higher. String Comparison and Text Similarity compare the output with a dataset column. Live traffic has no dataset, so these types run only in offline runs.

## Find an evaluator

The _Evaluators_ page lists every evaluator in your company. Go to _Agent Experimentation > Evaluators_ to open it.

The list has the columns **Name**, **Type**, **Last Modified**, and **Creator**. Search the list, filter it by **Creator**, or sort it by **Name** or **Last Modified**. Click an evaluator to open its page, which has three tabs: _Setup_, _Results_, and _Traces_.

## Create an evaluator

Start from a template when one matches your check. Start from scratch to write your own.

1. Go to _Agent Experimentation > Evaluators_ and click **Create**. If your company has no evaluators yet, click **Create an Evaluator**.
2. In the _Create Evaluator_ dialog, click a template under **Create from Template**, or click **Create from Scratch**.
3. Enter an **Evaluator Name**. Evaluator names must be unique in your company.
4. (Optional) Enter a description.
5. Click **Create**.

Statsig creates the evaluator and opens its _Setup_ tab. A template evaluator starts with the template's prompt or code saved. An evaluator created from scratch starts as an LLM as a Judge evaluator with an empty prompt.

You can also create an evaluator from a config. On the config's _Setup_ tab, in the _Run Setup_ section, click **Add evaluator**, then click **Create new evaluator**. Evaluators you create this way join the same catalog. For more about configs, go to [Configs](https://docs.statsig.com/agent-experimentation/configs).

### Evaluator templates

The _Create Evaluator_ dialog offers six templates. Each template is a starting point that you can edit after you create the evaluator.

| Template | Type | What it checks | Runs online |
| --- | --- | --- | --- |
| Helpfulness | LLM as a Judge | Whether the response resolves the request | Yes |
| Tone Check | LLM as a Judge | Whether the response is polite and on-brand. Edit the voice list in the prompt to match your brand. | Yes |
| Groundedness | LLM as a Judge | Whether the request or tool results back every claim | Yes |
| Faithfulness | LLM as a Judge | Whether the output contradicts the request or tool results | Yes |
| Exact Match | Python | Whether the output equals the value in a dataset column, `expected` by default | No, offline runs only |
| JSON Validity | Python | Whether the output parses as JSON, with optional required fields | Yes |

## Set up an evaluator

The _Setup_ tab holds the evaluator's definition and its targeting rules. It has these fields:

- **Name**: The evaluator's name in the catalog.
- **Type**: One of the four [evaluator types](#grader-types).
- **Evaluation Model**: The provider and model that score output. This field appears for LLM as a Judge evaluators only.
- **Prompt** or **Definition**: The judge prompt for LLM as a Judge evaluators, or the comparison or code for the other types.
- **Targeting rules**: Where the evaluator runs. For details, go to [Choose where an evaluator runs](#choose-where-a-grader-runs).

### Choose the evaluation model

An LLM as a Judge evaluator uses the model in **Evaluation Model** to score each row or span. Choose a **Provider**, such as OpenAI, Anthropic, or Google, and then a **Model**. New evaluators use OpenAI `gpt-5-mini`.

Judge calls use your company's Included Credits unless your company has an API key for that provider. Without a key for the provider, the _Setup_ tab shows a notice with an **Add API key** link. For details, go to [Evaluator costs](#grader-costs).

### Write a judge prompt

The judge prompt tells the evaluation model what to check and how to score it. Reference the output and its context with these variables. To insert a variable, click it in the _Variables_ list under the prompt.

| Variable | Value |
| --- | --- |
| `{{input}}` | The input messages, one `role: text` line for each message. In offline runs, the dataset's Input column. |
| `{{output}}` | The final answer text. |
| `{{trajectory}}` | The tool calls, tool results, and reasoning before the final answer. Empty when the output has none. |

Wrap a variable in three braces, such as `{{{output}}}`, to pass its text unchanged. With two braces, Statsig HTML-escapes the text, so the judge reads characters such as `"` and `<` as `&quot;` and `&lt;`. The templates use three braces. For example:

```text
Score how well the answer resolves the request, from 0 to 1.

<request>
{{{input}}}
</request>
<answer>
{{{output}}}
</answer>
```

A prompt can also reference a dataset column by name, such as `{{expected}}`. An evaluator whose prompt references any variable other than `{{input}}`, `{{output}}`, and `{{trajectory}}` can be used in offline runs only.

### Define a String Comparison or Text Similarity evaluator

String Comparison and Text Similarity evaluators compare `{{output}}` with a column from the dataset that a run uses.

- **String Comparison**: Choose an operation and the column to compare with. The operations are Equals, Does not equal, Contains, and Contains (ignore case).
- **Text Similarity**: Choose a function and the column to compare with. The functions are Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, and Cosine.

The **Column** list shows the columns of a config's dataset, so create these evaluators from the config that runs them. On the config's _Setup_ tab, click **Add evaluator**, then click **Create new evaluator**.

### Write a Python evaluator

A Python evaluator runs your code once for each row in an offline run, or once for each graded span in online evaluation. The code must define a `grade(sample, item)` function that returns a number. For example, this evaluator passes outputs of 500 characters or fewer:

```python
def grade(sample, item):
    output = sample["output_text"]
    return 1.0 if len(output) <= 500 else 0.0
```

| Part | Contract |
| --- | --- |
| Function | `grade(sample, item)`, defined at the top level of your code |
| Output | `sample["output_text"]` holds the output text. |
| Item | `item` holds the dataset row's columns by name. `item["input"]` and `item["trajectory"]` hold the input and trajectory in both offline runs and online evaluation. |
| Return value | An `int` or `float` that isn't `NaN` or infinite. A score of 0.67 or higher passes. |
| Errors | If the code has no `grade` function, raises an exception, or returns anything other than a finite number, the row scores 0. Statsig records the error message as the evaluator's response for that row. |
| Time limit | 10 seconds for each row. A row that runs longer scores 0. |
| Source limit | 100,000 characters. Code over the limit fails every row. |
| Environment | A sandbox with the Python standard library. Statsig clears environment variables. |

To run online, a Python evaluator can read only `item["input"]` and `item["trajectory"]` from `item`, written as literal keys. Code that reads any other key, or a key held in a variable, can be used in offline runs only.

### Save an evaluator version

Edits on the _Setup_ tab stay unsaved until you click **Save Evaluator**. Click **Discard Changes** to undo them.

Saving a change to the evaluator's definition creates a new version of the evaluator. When configs use the evaluator, Statsig first shows the _Update this evaluator?_ dialog. Saving updates the evaluator for the latest version of each config that uses it. Runs that already finished keep the evaluator version they used.

## Choose where an evaluator runs

Targeting rules decide which offline runs include an evaluator and which live traffic it grades online. Set them in the **Targeting rules** section of the evaluator's _Setup_ tab. Each rule is an offline rule, an online rule, or both.

To add a rule:

1. On the evaluator's _Setup_ tab, under **Targeting rules**, click **Add rule**.
2. Check **Offline**, **Online**, or both.
3. Set the rule's conditions. For each condition, choose a field, an operator, and one or more values.
4. For an online rule, enter the percentage of matching traffic to grade in **Sample**.
5. Click **Save Evaluator**.

A rule can be both offline and online when its only condition is "Config is any of." The **Sample** percentage then applies to online grading only, because offline runs grade every row.

### Offline rules

An offline rule adds the evaluator to the runs of configs. It takes exactly one condition, "Config is any of," with no sampling. Check **Critical** to make the evaluator a [critical evaluator](#what-is-a-critical-grader) for these configs. On each config's _Setup_ tab, the evaluator card shows an Offline badge.

Adding an evaluator from a config's _Setup_ tab with **Add evaluator** has the same effect as a "Config is any of" rule. A run of a config uses at most 10 evaluators.

Runs on the _Runs_ page don't use offline rules. Each run uses the evaluators you pick when you create the run.

### Online rules

An online rule grades a sample of live traffic as it arrives. Only evaluators that can run online, as listed in [Evaluator types](#grader-types), can have online rules.

Online evaluation grades one span for each agent turn: the workflow span when the trace has one, or else the outermost LLM call. The span must carry input and output messages, and tool spans are never graded. Spans don't need a config, so a rule can also grade traced LLM calls that your app makes without one. For how to send traces, go to [Online Evaluation](https://docs.statsig.com/agent-experimentation/online-evaluation).

An online rule matches a span when all of its conditions match. A rule with no conditions matches every span. Each condition uses one of these fields:

| Field | Matches |
| --- | --- |
| Config | The span's config, from `statsig.llm.prompt_name` |
| Environment | `statsig.environment` |
| Span name | The span's name |
| Model | The requested model, from `gen_ai.request.model` |
| Span attribute | Any span attribute. Enter the attribute name, then the values. |

Each condition uses the operator **is any of** or **is not**. Values match exactly, with no wildcards, and can't contain `%`. A span that doesn't have the attribute counts as an empty value, so **is not** matches it.

Each evaluator grades a span at most once, under the first enabled online rule that matches the span. Use the arrows on a rule to move it up or down. **Sample** sets the percentage of matching traces that the rule grades, from 0 to 100. New rules start at 10%. Statsig samples by trace: it hashes the trace ID, so every span in a trace gets the same sampling decision.

Online grading has these limits:

- Statsig grades at most 1,000 spans per hour for each config. A span counts toward its config's limit when every rule that grades it has a "Config is any of" condition that includes the config. Other spans that online rules grade share one limit of 1,000 spans per hour for your company.
- Statsig processes traces for online evaluation in the US only. Statsig plans to support EU data residency.

### Grade a config's production traffic

Online rules alone decide which of a config's spans Statsig grades. To grade a config's spans, save an enabled online rule with a "Config is any of" condition that includes the config. To stop, disable the rule or remove the config from the condition. Rules that don't name the config can also grade its spans when their conditions match.

### Disable an online rule

To stop an online rule without deleting it, clear its **Online** checkbox and save. Statsig keeps the rule, and the evaluator cards on configs show it with a Disabled badge.

### Who can edit targeting rules

Enabled online rules start grading live traffic, so a project or org admin must save any change to them. This includes adding, editing, disabling, or deleting an enabled online rule. Anyone who can edit the evaluator can change its offline rules and its disabled online rules.

### Targeting rule errors

**Save Evaluator** stays disabled until every rule is valid. These messages explain what to fix:

| Message | Fix |
| --- | --- |
| Check Offline, Online or both | Check at least one of **Offline** and **Online**. |
| Offline rules need one "Config is any of" condition | Reduce the offline rule to one "Config is any of" condition. |
| Every condition needs at least one value | Add a value to each condition, or remove the empty condition. |
| Span attribute conditions need an attribute name | Enter the attribute name in the Span attribute condition. |
| Condition values cannot contain % | Remove `%` from the value. Values match exactly. |
| Sampling rate must be between 0% and 100% | Enter a **Sample** value from 0 to 100. |
| Online rules need an evaluator that reads just `{{input}}`, `{{output}}` and `{{trajectory}}` | Remove the online rules, or change the definition so the evaluator can run online. |

## What is a primary evaluator

A primary evaluator is the evaluator whose score becomes a config version's overall score, the number you compare across versions. Each config has one primary evaluator. To change it, open the menu on an evaluator card in the config's _Run Setup_ section and click **Set as Primary**. If you remove the primary evaluator, the next evaluator becomes primary.

## What is a critical evaluator

A critical evaluator is a must-pass check for a config's runs. When a critical evaluator's score for a run is 0, the whole run fails. The run shows "This run failed due to the critical evaluator" with the evaluator's name, and the version's overall score is 0. When a critical evaluator scores above 0, it behaves like any other evaluator.

For example, if your app parses a config's output as JSON, mark a JSON Validity evaluator critical for that config. A version whose output never parses then fails its run, whatever its other evaluators score.

You set Critical separately for each config. Make an evaluator critical for a config in either place:

- On the config's _Setup_ tab, open the menu on the evaluator card and click **Set as Critical Evaluator**.
- On the evaluator's _Setup_ tab, check **Critical** on an offline rule with a "Config is any of" condition.

Critical grading applies to offline runs only.

## Review an evaluator's results

The evaluator's _Results_ and _Traces_ tabs show where it runs and how it scores.

- **Results**: The _Used by_ table lists each config and Playground that uses the evaluator. Its Type column shows Config or Playground. The _Runs_ table lists each run group that ran the evaluator, with columns for Score, Run Group, Type, Config, and Last run. Type is Online for online evaluation and On Demand for all other run groups. An evaluator that nothing uses shows "This evaluator is currently unused."
- **Traces**: The traces that the evaluator scored, from the last day by default. Change the time range or add filters, and click a trace to open its details. Statsig tracks scores by evaluator ID, so renaming an evaluator keeps its history.

## Evaluator costs

LLM as a Judge evaluators and Text Similarity evaluators that use Cosine call a model provider. Statsig charges these calls to your Included Credits unless your company has its own API key for the provider. Other evaluators run inside Statsig at no cost.

| Evaluator | Provider it calls | Cost |
| --- | --- | --- |
| LLM as a Judge | The provider in **Evaluation Model** | Included Credits, or your own key for that provider |
| Text Similarity with Cosine | OpenAI, for `text-embedding-3-large` embeddings | Included Credits, or your own OpenAI key |
| String Comparison, Python, and other Text Similarity functions | None | Free |

Statsig doesn't meter calls on your own API keys. An admin adds keys in _Settings > Product Configuration > General_, under **AI Provider API Keys**. When your Included Credits run out, online evaluation keeps running the evaluators that use your own keys and skips the rest. For details on Included Credits, go to [Credits and API keys](https://docs.statsig.com/agent-experimentation/overview#credits-and-api-keys).