---
title: Online Evaluation
description: "Grade production traffic in Statsig Agent Experimentation with online evaluators, which score a sample of agent turns about a minute after your app sends their spans."
product: general
lang: en
token_estimate: 4430
---
# Online Evaluation

Online evaluation grades your AI app's production traffic. Statsig samples the traces your app sends and picks one span for each agent turn. About one minute after that span arrives, Statsig runs your online evaluators on it. Online evaluators score each answer on its own, without an expected answer to compare against. Use online evaluation to compare the Live version of a [config](https://docs.statsig.com/agent-experimentation/configs) with its Candidate versions on real requests. To score versions against a dataset of expected answers before you release them, use [offline evaluation](https://docs.statsig.com/agent-experimentation/offline-evaluation) instead.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

## How online evaluation works

Online evaluation has four stages:

1. Your app answers each request with the config's Live version. It can also run Candidate versions on the same input without showing their answers to users. Your code runs the candidates. Statsig doesn't run them for you.
2. Your app sends a span for each model call to Statsig, tagged with the config and version that produced it.
3. Each evaluator's online rules decide which spans the evaluator scores and what percentage of traces it samples.
4. Statsig grades each sampled span once. Scores appear on the span and on the config's _Traces_ tab.

Your code can load a config's Live version and its Candidate versions. Draft and Archived versions never reach your code. To manage versions, go to [Configs](https://docs.statsig.com/agent-experimentation/configs).

## Instrument your app

Online grading reads the spans your app sends. Each span needs its input and output messages, its config, and its version. With the [Node AI SDK](https://docs.statsig.com/agent-experimentation/node), do the following:

- Wrap your model client with `wrapOpenAI`, `wrapAnthropic`, or `wrapGemini`, and turn on `captureOptions` for input and output messages. The SDK doesn't record message content by default, and Statsig skips spans without messages.
- Call the model inside `prompt.withLive()` or `prompt.withVersion()`. Both functions tag each span with the config name and version. A span without a config doesn't show on any config's _Traces_ tab.
- Wrap each multi-step agent turn in one workflow span with `wrap({ type: "workflow" })`.

The SDK still calls a config a prompt, so you load one with `getPrompt()`. To send spans without the SDK, set the attributes in the [span attribute reference](https://docs.statsig.com/agent-experimentation/traces#span-attribute-reference) on each span yourself.

The examples on this page share this setup:

```js
import OpenAI from "openai";
import { StatsigAI, initializeTracing, wrapOpenAI } from "@statsig/statsig-ai";

// Sends spans to Statsig with the server secret key in STATSIG_SDK_KEY
initializeTracing({ serviceName: "support-bot" });

const statsigAI = new StatsigAI({ sdkKey: process.env.STATSIG_SDK_KEY });
await statsigAI.initialize();

// Record the input and output messages that online evaluators read
const openai = wrapOpenAI(new OpenAI(), {
  captureOptions: {
    capture_input_messages: true,
    capture_output_messages: true,
  },
});
```

### Grade a single model call

When an agent turn makes one model call, Statsig grades that call. The SDK adds the `statsig.span.llm_root` marker to each model call that isn't inside another model call or a workflow span. An HTTP span or another app span around the call doesn't change the marker. Statsig grades the spans that carry this marker.

The following example answers with the Live version. It also runs each Candidate version on the same question in the background. Each call produces its own span, tagged with its version, so Statsig grades each version separately.

```js
// Calls the model with one config version
function callModel(version, question) {
  return openai.chat.completions.create({
    model: version.getModel({ fallback: "gpt-5-mini" }),
    messages: version.getPromptMessages({ question }),
  });
}

async function answerQuestion(user, question) {
  const prompt = statsigAI.getPrompt(user, "support_bot");

  // Candidates answer in the background. Users never see these answers.
  for (const candidate of prompt.getCandidates()) {
    prompt
      .withVersion(candidate, (version) => callModel(version, question))
      .catch(console.error);
  }

  const response = await prompt.withLive((version) =>
    callModel(version, question),
  );
  return response.choices[0].message.content;
}
```

### Grade a multi-step agent turn

When one turn makes several model calls or tool calls, wrap the turn in a workflow span. The workflow span ends after every step in the turn, so it can hold the turn's input and final answer. Statsig grades the workflow span as the turn's answer. Statsig never grades tool spans, such as spans from `wrap({ type: "tool" })`.

The SDK doesn't record messages on workflow spans. Before the workflow function returns, set `gen_ai.input.messages` and `gen_ai.output.messages` on the active span to the turn's input and final answer. Without these attributes, Statsig skips the workflow span. The SDK doesn't mark the model calls inside a workflow span, so Statsig then grades nothing in the turn. Start the turn inside `withLive()` or `withVersion()` so the workflow span records the config and version.

```js
import { trace } from "@opentelemetry/api";
import { wrap } from "@statsig/statsig-ai";

// One workflow span for each agent turn
const runTurn = wrap(
  { type: "workflow", name: "support_turn" },
  async (version, question) => {
    const model = version.getModel({ fallback: "gpt-5-mini" });

    // Step 1: draft an answer from the config's prompt
    const draft = await openai.chat.completions.create({
      model,
      messages: version.getPromptMessages({ question }),
    });

    // Step 2: revise the draft
    const revision = await openai.chat.completions.create({
      model,
      messages: [
        {
          role: "system",
          content: "Correct any errors in this answer and shorten it.",
        },
        { role: "user", content: draft.choices[0].message.content ?? "" },
      ],
    });
    const answer = revision.choices[0].message.content ?? "";

    // The SDK doesn't record messages on workflow spans, so set the turn's
    // question and final answer for online evaluators to read
    trace.getActiveSpan()?.setAttributes({
      "gen_ai.input.messages": JSON.stringify([
        { role: "user", content: question },
      ]),
      "gen_ai.output.messages": JSON.stringify([
        { role: "assistant", content: answer },
      ]),
    });
    return answer;
  },
);

async function answerWithAgent(user, question) {
  const prompt = statsigAI.getPrompt(user, "support_bot");
  // Start the turn inside withLive so the workflow span records the config and version
  return prompt.withLive((version) => runTurn(version, question));
}
```

> **Warning:**
>
> In `@statsig/statsig-ai` versions `0.0.1-beta.30` through `0.0.1-beta.33`, the SDK also marks each model call inside a workflow span with `statsig.span.llm_root`. Statsig checks each span on its own, so it grades the workflow span and each model call inside it. Version `0.0.1-beta.34` doesn't mark these calls. If you can't upgrade, add a **Span attribute** condition to the evaluator's online rule to grade only workflow spans. Set the attribute name to `statsig.span.gen_ai.type`, the operator to **is any of**, and the value to `workflow`. A rule with this condition doesn't grade turns that make a single model call.

## Turn on online grading

Online grading starts when an evaluator has an enabled online rule that matches your spans.

1. Open an evaluator that can run online, or create one. LLM as a Judge and Python evaluators can run online when they read only `{{input}}`, `{{output}}`, and `{{trajectory}}`. For details, go to [Evaluator types](https://docs.statsig.com/agent-experimentation/evaluators#grader-types).
2. On the evaluator's _Setup_ tab, under **Targeting rules**, click **Add rule** and check **Online**.
3. Add conditions that match your spans. For example, choose **Config**, choose **is any of**, and select your config.
4. In **Sample**, enter the percentage of matching traces to grade. New rules start at 10%.
5. Click **Save Evaluator**. A project or org admin must save any change to an enabled online rule.

For every rule field and operator, go to [Online rules](https://docs.statsig.com/agent-experimentation/evaluators#online-rules).

Online rules choose spans in these ways:

- Each evaluator grades a span under the first enabled online rule that matches the span.
- Statsig samples by trace. Statsig hashes the trace ID, so every span in a trace gets the same sampling decision.
- To stop grading a config's spans, [disable](https://docs.statsig.com/agent-experimentation/evaluators#disable-an-online-rule) the rules that name the config, or remove the config from their conditions. Rules that don't name the config keep grading its spans.
- A rule without a **Config** condition can also grade spans that don't belong to any config, such as model calls outside `withLive()` and `withVersion()`. Scores for these spans appear in the trace drawer and on the evaluator's _Traces_ tab, but not on a config.

## What Statsig grades

Statsig checks each span when it arrives. Statsig grades a span when every requirement in this table holds:

| Requirement | Span attributes |
| --- | --- |
| The span is an LLM span. | `statsig.span.type` is `gen_ai`. |
| The span is a workflow span or an outermost model call. Statsig never grades tool spans. | `statsig.span.gen_ai.type` is `workflow`, or `statsig.span.llm_root` is `"1"` and `statsig.span.gen_ai.type` isn't `tool`. |
| The span has input and output messages. | `gen_ai.input.messages` and `gen_ai.output.messages` aren't empty. |
| Statsig didn't generate the span. | `statsig.environment` isn't `eval`, `shadow`, or `online_eval_judge`. Statsig treats a span without `statsig.environment` as production. |
| An online rule matches the span and samples its trace. | The attributes in the rule's conditions. |

Statsig assigns each score to the config version in `statsig.llm.prompt_version_id`. For spans from older SDKs that record only the version name, Statsig finds the version by its name.

An evaluator reads three values from the span. `{{input}}` holds the input messages, `{{output}}` holds the final assistant answer, and `{{trajectory}}` holds the tool calls and results before that answer. For message formats and every attribute, go to the [span attribute reference](https://docs.statsig.com/agent-experimentation/traces#span-attribute-reference). For the environment values, go to [Environments](https://docs.statsig.com/agent-experimentation/traces#environments).

## Online grading limits

Online grading has these limits:

- **Latency**: A score appears on its span about one minute after Statsig receives the span. The Online evaluators chart can take longer to show the score.
- **Hourly cap**: Statsig grades at most 1,000 spans per hour for each config. A span counts toward its config's cap when every rule that grades it has a "Config is any of" condition that includes the config. All other graded spans, including spans without a config, share one cap of 1,000 spans per hour for your company. Statsig doesn't grade spans over a cap, and the trace drawer shows `No evaluator scores this span.`
- **One grade per span**: Each evaluator grades a span at most once. Statsig retries failed judge calls. If an evaluator still fails, the span has no score from that evaluator.
- **Included Credits**: When your company's Included Credits run out, Statsig keeps running the evaluators that use your own provider API key. Statsig skips the other evaluators. The trace drawer explains that Statsig didn't grade the span. An admin adds API keys in _Settings > Product Configuration > General_, under **AI Provider API Keys**. For which evaluators use credits, go to [Evaluator costs](https://docs.statsig.com/agent-experimentation/evaluators#grader-costs). For how Included Credits work, go to [Credits and API keys](https://docs.statsig.com/agent-experimentation/overview#credits-and-api-keys).
- **Data residency**: Statsig processes online evaluation traces in the US only. Statsig plans to support EU data residency.

## Review online evaluation results

Statsig shows online grades on the config's _Traces_ tab, on each span, and on each evaluator's page.

### Online evaluators chart

To open the chart, go to _Agent Experimentation > Configs_, open the config, and click the **Traces** tab. The Online evaluators chart appears above the span list and covers the last day by default. Choose a view:

- **Single version**: Each evaluator's average score over time, for the version you select or for **All versions**. Grades that your code logs with `logEvalGrade` appear as separate lines, labeled with the grade name and "(SDK)".
- **Compare**: Select the versions to compare with the Live version. For each evaluator, the chart shows the score change relative to the Live version. The legend marks each change as Better, Worse, or No significant change.

### How Statsig compares versions

The Compare view tests each evaluator's score change for statistical significance at p < 0.05. The test depends on where the scores come from:

| Score source | Comparison |
| --- | --- |
| Online evaluators | Welch's t-test, unpaired. Each graded span is one sample, because each trace comes from one version. Each version needs at least two scores. |
| `logEvalGrade` (SDK) | Paired by `sessionId`. Statsig compares only sessions that have a grade for both versions. |
| Offline runs | Not included. Offline runs grade dataset rows, not production traffic. |

### Scores on a span

To open a span's scores, click the span in the list, then click the **Evaluators** tab in the trace drawer. Each score shows its evaluator and its score. Online grades show the evaluator type, such as LLM as a Judge or Python. Grades from `logEvalGrade` show SDK. For the full tab, go to [Evaluators tab](https://docs.statsig.com/agent-experimentation/traces#graders-tab).

An evaluator's _Results_ and _Traces_ tabs also list its online grades. Statsig tracks scores by evaluator ID, so a renamed evaluator keeps its history. For details, go to [Review an evaluator's results](https://docs.statsig.com/agent-experimentation/evaluators#review-a-graders-results).

## Log grades from your code

Use `logEvalGrade` to record a score that your own code computes, such as a user's rating or a business outcome. Statsig shows these grades as SDK lines on the Online evaluators chart and as SDK scores in the trace drawer.

```js
statsigAI.logEvalGrade(user, version, 1, "ticket_resolved", {
  sessionId: conversationId,
  traceId,
  spanId,
  reasoning: "The customer closed the ticket without escalating.",
  metadata: { channel: "chat" },
});

// Send the grade events to Statsig
await statsigAI.flushEvents();
```

`logEvalGrade` takes these arguments, in order:

| Argument | Description |
| --- | --- |
| `user` | The `StatsigUser` for the request. |
| `version` | The config version that produced the answer, such as the version that `withLive()` passes to your function. |
| `score` | A number from 0 to 1. The SDK skips scores outside this range and logs a warning. |
| `graderName` | The name of the grade, such as `ticket_resolved`. |
| `evalData` | An object with optional fields: `sessionId`, `activityId`, `traceId`, `spanId`, `reasoning`, and `metadata`. Always pass this argument. Pass `{}` when you set no fields. |

The `evalData` object accepts these fields:

| Field | Description |
| --- | --- |
| `sessionId` | Pairs grades across versions in the Compare view. Use the same ID for the live and candidate answers to one request. |
| `activityId` | Adds a `statsig.activity_id` custom ID to the user. |
| `traceId`, `spanId` | The span that the grade belongs to in the trace drawer. When you omit both, the SDK uses the active span. |
| `reasoning` | Your explanation of the score, logged with the grade. |
| `metadata` | Extra string fields to log with the grade. |

To grade a workflow span after it ends, such as when a user rates an answer later, save the span's IDs during the turn. Inside the workflow function, `trace.getActiveSpan()?.spanContext()` returns the workflow span's `traceId` and `spanId`.

## Grade events in your project

Statsig logs each online grade as a `statsig::eval_result` event in your project. The Online evaluators chart reads these events. The event's value is the config name, or empty for a span without a config. The events have no user ID.

| Metadata field | Value |
| --- | --- |
| `score` | The score, from 0 to 1. |
| `version_id`, `version_name` | The config version that produced the span. |
| `grader_id`, `grader_name`, `grader_version` | The evaluator and the evaluator version that scored the span. |
| `ai_config_name` | The config name. |
| `source` | `online_eval`. |
| `trace_id`, `span_id` | The graded span. |

Statsig logs these events with a server secret key that has the description `AI_EVALS`. Statsig creates the key in your project the first time it logs an online grade.

`logEvalGrade` logs the same event name with `source` set to `sdk`. Its events also carry `session_id`, `is_live`, and the fields you pass in `metadata`. Evaluator metrics in experiments that you create from a config read only grades from `logEvalGrade`. Online grades don't feed those metrics.