---
title: Graders
description: "Create and configure graders in Statsig Agent Experimentation to score AI output in offline and online evals, with LLM judges, Python, string, and similarity checks."
product: general
lang: en
token_estimate: 4466
---
# Graders

A grader scores AI output with a number from 0 to 1. Statsig runs graders in two places: offline evals, which grade every row of a dataset, and online evals, which grade a sample of live traffic. Graders live in one catalog for your company. Any AI Config or eval can reuse a grader, so its scores stay comparable across versions.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Use a grader when you want Statsig to compute the score. To record a score that your own code computes, such as user feedback, call `logEvalGrade` from the [Node AI SDK](https://docs.statsig.com/ai-evals/node#logging-eval-results) instead.

## Grader types

Every grader has one of four types. The type decides how the grader scores output and whether it can run in online evals.

| Type | How it scores | Runs online |
| --- | --- | --- |
| LLM as a Judge | A model reads your prompt and the output, then returns a score from 0 to 1 with its reasoning. | Yes, when the prompt uses only `{{input}}`, `{{output}}`, and `{{trajectory}}` |
| String Comparison | Compares `{{output}}` with a dataset column using Equals, Does not equal, Contains, or Contains (ignore case). Scores 1 when the comparison holds and 0 when it doesn't. | No |
| Text Similarity | Scores how similar `{{output}}` is to a dataset column, using Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, or Cosine. Cosine embeds both texts with the OpenAI `text-embedding-3-large` model. | No |
| Python | Runs your `grade(sample, item)` function, which returns a number. | Yes, when the code reads only `item["input"]` and `item["trajectory"]` |

A row or span passes a grader when it scores 0.67 or higher. String Comparison and Text Similarity compare the output with a dataset column. Live traffic has no dataset, so these types run only in offline evals.

## Find a grader

The Graders page lists every grader in your company. Go to _Agent Experimentation > Graders_ to open it.

The list has the columns **Name**, **Type**, **Last Modified**, and **Creator**. Search the list, filter it by **Creator**, or sort it by **Name** or **Last Modified**. Click a grader to open its page, which has three tabs: _Setup_, _Results_, and _Traces_.

## Create a grader

Start from a template when one matches your check. Start from scratch to write your own.

1. Go to _Agent Experimentation > Graders_ and click **Create**. If your company has no graders yet, click **Create a Grader**.
2. In the _Create Grader_ dialog, click a template under **Create from Template**, or click **Create from Scratch**.
3. Enter a **Grader Name**. Grader names must be unique in your company.
4. (Optional) Enter a description.
5. Click **Create**.

Statsig creates the grader and opens its _Setup_ tab. A template grader starts with the template's prompt or code saved. A grader created from scratch starts as an LLM as a Judge grader with an empty prompt.

You can also create a grader from an AI Config. On the AI Config's _Setup_ tab, in the _Eval Setup_ section, click **Add grader**, then click **Create new grader**. Graders you create this way join the same catalog. For more about AI Configs, go to [AI Configs](https://docs.statsig.com/ai-evals/prompts).

### Grader templates

The _Create Grader_ dialog offers six templates. Each template is a starting point that you can edit after you create the grader.

| Template | Type | What it checks | Runs online |
| --- | --- | --- | --- |
| Helpfulness | LLM as a Judge | Whether the response resolves the request | Yes |
| Tone Check | LLM as a Judge | Whether the response is polite and on-brand. Edit the voice list in the prompt to match your brand. | Yes |
| Groundedness | LLM as a Judge | Whether the request or tool results back every claim | Yes |
| Faithfulness | LLM as a Judge | Whether the output contradicts the request or tool results | Yes |
| Exact Match | Python | Whether the output equals the value in a dataset column, `expected` by default | No, dataset evals only |
| JSON Validity | Python | Whether the output parses as JSON, with optional required fields | Yes |

## Set up a grader

The _Setup_ tab holds the grader's definition and its targeting rules. It has these fields:

- **Name**: The grader's name in the catalog.
- **Type**: One of the four [grader types](#grader-types).
- **Evaluation Model**: The provider and model that score output. This field appears for LLM as a Judge graders only.
- **Prompt** or **Definition**: The judge prompt for LLM as a Judge graders, or the comparison or code for the other types.
- **Targeting rules**: Where the grader runs. For details, go to [Choose where a grader runs](#choose-where-a-grader-runs).

### Choose the evaluation model

An LLM as a Judge grader uses the model in **Evaluation Model** to score each row or span. Choose a **Provider**, such as OpenAI, Anthropic, or Google, and then a **Model**. New graders use OpenAI `gpt-5-mini`.

Judge calls use your company's Included Credits unless your company has an API key for that provider. Without a key for the provider, the _Setup_ tab shows a notice with an **Add API key** link. For details, go to [Grader costs](#grader-costs).

### Write a judge prompt

The judge prompt tells the evaluation model what to check and how to score it. Reference the output and its context with these variables. To insert a variable, click it in the _Variables_ list under the prompt.

| Variable | Value |
| --- | --- |
| `{{input}}` | The input messages, one `role: text` line for each message. In offline evals, the dataset's Input column. |
| `{{output}}` | The final answer text. |
| `{{trajectory}}` | The tool calls, tool results, and reasoning before the final answer. Empty when the output has none. |

Wrap a variable in three braces, such as `{{{output}}}`, to pass its text unchanged. With two braces, Statsig HTML-escapes the text, so the judge reads characters such as `"` and `<` as `&quot;` and `&lt;`. The templates use three braces. For example:

```text
Score how well the answer resolves the request, from 0 to 1.

<request>
{{{input}}}
</request>
<answer>
{{{output}}}
</answer>
```

A prompt can also reference a dataset column by name, such as `{{expected}}`. A grader whose prompt references any variable other than `{{input}}`, `{{output}}`, and `{{trajectory}}` runs in offline evals only.

### Define a String Comparison or Text Similarity grader

String Comparison and Text Similarity graders compare `{{output}}` with a column from the dataset that an eval runs on.

- **String Comparison**: Choose an operation and the column to compare with. The operations are Equals, Does not equal, Contains, and Contains (ignore case).
- **Text Similarity**: Choose a function and the column to compare with. The functions are Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, and Cosine.

The **Column** list shows the columns of an AI Config's dataset, so create these graders from the AI Config that runs them. On the AI Config's _Setup_ tab, click **Add grader**, then click **Create new grader**.

### Write a Python grader

A Python grader runs your code once for each row in an offline eval, or once for each graded span in an online eval. The code must define a `grade(sample, item)` function that returns a number. For example, this grader passes outputs of 500 characters or fewer:

```python
def grade(sample, item):
    output = sample["output_text"]
    return 1.0 if len(output) <= 500 else 0.0
```

| Part | Contract |
| --- | --- |
| Function | `grade(sample, item)`, defined at the top level of your code |
| Output | `sample["output_text"]` holds the output text. |
| Item | `item` holds the dataset row's columns by name. `item["input"]` and `item["trajectory"]` hold the input and trajectory in both offline and online evals. |
| Return value | An `int` or `float` that isn't `NaN` or infinite. A score of 0.67 or higher passes. |
| Errors | If the code has no `grade` function, raises an exception, or returns anything other than a finite number, the row scores 0. Statsig records the error message as the grader's response for that row. |
| Time limit | 10 seconds for each row. A row that runs longer scores 0. |
| Source limit | 100,000 characters. Code over the limit fails every row. |
| Environment | A sandbox with the Python standard library. Statsig clears environment variables. |

To run online, a Python grader can read only `item["input"]` and `item["trajectory"]` from `item`, written as literal keys. Code that reads any other key, or a key held in a variable, runs in offline evals only.

### Save a grader version

Edits on the _Setup_ tab stay unsaved until you click **Save Grader**. Click **Discard Changes** to undo them.

Saving a change to the grader's definition creates a new version of the grader. When AI Configs use the grader, Statsig first shows the _Update this grader?_ dialog, which calls AI Configs "prompts." Saving updates the grader for the latest version of each AI Config that uses it. Eval runs that already finished keep the grader version they used.

## Choose where a grader runs

Targeting rules decide which offline evals include a grader and which live traffic it grades online. Set them in the **Targeting rules** section of the grader's _Setup_ tab. Each rule is an offline rule, an online rule, or both.

To add a rule:

1. On the grader's _Setup_ tab, under **Targeting rules**, click **Add rule**.
2. Check **Offline**, **Online**, or both.
3. Set the rule's conditions. For each condition, choose a field, an operator, and one or more values.
4. For an online rule, enter the percentage of matching traffic to grade in **Sample**.
5. Click **Save Grader**.

A rule can be both offline and online when its only condition is "AI config is any of." The **Sample** percentage then applies to online grading only, because offline evals grade every row.

### Offline rules

An offline rule adds the grader to the version evals of AI Configs. It takes exactly one condition, "AI config is any of," with no sampling. Check **Critical** to make the grader a [critical grader](#what-is-a-critical-grader) for these AI Configs. On each AI Config's _Setup_ tab, the grader card shows an Offline badge.

Adding a grader from an AI Config's _Setup_ tab with **Add grader** has the same effect as an "AI config is any of" rule. A version eval runs at most 10 graders.

Runs on the _Evals_ page don't use offline rules. Each run uses the graders you pick when you create the run.

### Online rules

An online rule grades a sample of live traffic as it arrives. Only graders that can run online, as listed in [Grader types](#grader-types), can have online rules.

Online evals grade one span for each agent turn: the workflow span when the trace has one, or else the outermost LLM call. The span must carry input and output messages, and tool spans are never graded. Spans don't need an AI Config, so a rule can also grade traced LLM calls that your app makes without one. For how to send traces, go to [Online Evals](https://docs.statsig.com/ai-evals/online-evals).

An online rule matches a span when all of its conditions match. A rule with no conditions matches every span. Each condition uses one of these fields:

| Field | Matches |
| --- | --- |
| AI config | The span's AI Config, from `statsig.llm.prompt_name` |
| Environment | `statsig.environment` |
| Span name | The span's name |
| Model | The requested model, from `gen_ai.request.model` |
| Span attribute | Any span attribute. Enter the attribute name, then the values. |

Each condition uses the operator **is any of** or **is not**. Values match exactly, with no wildcards, and can't contain `%`. A span that doesn't have the attribute counts as an empty value, so **is not** matches it.

Each grader grades a span at most once, under the first enabled online rule that matches the span. Use the arrows on a rule to move it up or down. **Sample** sets the percentage of matching traces that the rule grades, from 0 to 100. New rules start at 10%. Statsig samples by trace: it hashes the trace ID, so every span in a trace gets the same sampling decision.

Online grading has these limits:

- Statsig grades at most 1,000 spans per hour for each AI Config that has online eval turned on. Other spans that online rules grade share one limit of 1,000 spans per hour for your company.
- Statsig processes online-eval traces in the US only. Statsig plans to support EU data residency.

### Turn on online eval for an AI Config

A rule with an "AI config is any of" condition grades that AI Config's spans only while online eval is on for the AI Config. To turn it on, an admin opens the actions menu in the AI Config's header and clicks **Enable Online Eval**. The AI Config needs at least one saved online grader first. **Pause Online Eval** in the same menu turns online eval off.

While online eval is off for an AI Config, each rule that names the AI Config skips the AI Config's spans. Statsig checks those spans against the grader's later rules instead. The rule shows a warning that online evals are off for the AI Config, and the AI Config's grader card shows a Paused badge. Rules that don't name the AI Config keep grading its spans.

### Disable an online rule

To stop an online rule without deleting it, clear its **Online** checkbox and save. Statsig keeps the rule, and AI Config grader cards show it with a Disabled badge.

### Who can edit targeting rules

Enabled online rules start grading live traffic, so a project or org admin must save any change to them. This includes adding, editing, disabling, or deleting an enabled online rule. Anyone who can edit the grader can change its offline rules and its disabled online rules.

### Targeting rule errors

**Save Grader** stays disabled until every rule is valid. These messages explain what to fix:

| Message | Fix |
| --- | --- |
| Check Offline, Online or both | Check at least one of **Offline** and **Online**. |
| Offline rules need one "AI config is any of" condition | Reduce the offline rule to one "AI config is any of" condition. |
| Every condition needs at least one value | Add a value to each condition, or remove the empty condition. |
| Span attribute conditions need an attribute name | Enter the attribute name in the Span attribute condition. |
| Condition values cannot contain % | Remove `%` from the value. Values match exactly. |
| Sampling rate must be between 0% and 100% | Enter a **Sample** value from 0 to 100. |
| Online rules need a grader that reads just `{{input}}`, `{{output}}` and `{{trajectory}}` | Remove the online rules, or change the definition so the grader can run online. |

## What is a primary grader

A primary grader is the grader whose score becomes an AI Config version's overall score, the number you compare across versions. Each AI Config has one primary grader. To change it, open the menu on a grader card in the AI Config's _Eval Setup_ section and click **Set as Primary**. If you remove the primary grader, the next grader becomes primary.

## What is a critical grader

A critical grader is a must-pass check for an AI Config's version evals. When a critical grader's score for a run is 0, the whole run fails. The run shows "This eval run failed due to the critical grader" with the grader's name, and the version's overall score is 0. When a critical grader scores above 0, it behaves like any other grader.

For example, if your app parses an AI Config's output as JSON, mark a JSON Validity grader critical for that AI Config. A version whose output never parses then fails its eval, whatever its other graders score.

You set Critical separately for each AI Config. Make a grader critical for an AI Config in either place:

- On the AI Config's _Setup_ tab, open the menu on the grader card and click **Set as Critical Grader**.
- On the grader's _Setup_ tab, check **Critical** on an offline rule with an "AI config is any of" condition.

Critical grading applies to offline evals only.

## Review a grader's results

The grader's _Results_ and _Traces_ tabs show where it runs and how it scores.

- **Results**: The _Used by_ table lists each AI Config and Playground that uses the grader. Its Type column shows Prompt for an AI Config. The _Eval runs_ table lists each eval that ran the grader, with columns for Score, Eval, Type, Config, and Last run. Type is Online for online evals and On Demand for all other evals. A grader that nothing uses shows "This grader is currently unused."
- **Traces**: The AI traces that the grader scored, from the last day by default. Change the time range or add filters, and click a trace to open its details. Statsig tracks scores by grader ID, so renaming a grader keeps its history.

## Grader costs

LLM as a Judge graders and Text Similarity graders that use Cosine call a model provider. Statsig charges these calls to your Included Credits unless your company has its own API key for the provider. Other graders run inside Statsig at no cost.

| Grader | Provider it calls | Cost |
| --- | --- | --- |
| LLM as a Judge | The provider in **Evaluation Model** | Included Credits, or your own key for that provider |
| Text Similarity with Cosine | OpenAI, for `text-embedding-3-large` embeddings | Included Credits, or your own OpenAI key |
| String Comparison, Python, and other Text Similarity functions | None | Free |

Statsig doesn't meter calls on your own API keys. An admin adds keys in _Settings > Products_, under **AI Provider API Keys**. When your Included Credits run out, online evals keep running the graders that use your own keys and skip the rest. For details on Included Credits, go to [Credits and API keys](https://docs.statsig.com/ai-evals/overview#credits-and-api-keys).