Online Evaluation

Online evaluation grades your AI app's production traffic. Statsig samples the traces your app sends and picks one span for each agent turn. About one minute after that span arrives, Statsig runs your online evaluators on it. Online evaluators score each answer on its own, without an expected answer to compare against. Use online evaluation to compare the Live version of a config with its Candidate versions on real requests. To score versions against a dataset of expected answers before you release them, use offline evaluation instead.

How online evaluation works

Online evaluation has four stages:

1. Your app answers each request with the config's Live version. It can also run Candidate versions on the same input without showing their answers to users. Your code runs the candidates. Statsig doesn't run them for you.
2. Your app sends a span for each model call to Statsig, tagged with the config and version that produced it.
3. Each evaluator's online rules decide which spans the evaluator scores and what percentage of traces it samples.
4. Statsig grades each sampled span once. Scores appear on the span and on the config's Traces tab.

Your code can load a config's Live version and its Candidate versions. Draft and Archived versions never reach your code. To manage versions, go to Configs.

Instrument your app

Online grading reads the spans your app sends. Each span needs its input and output messages, its config, and its version. With the Node AI SDK, do the following:

• Wrap your model client with wrapOpenAI, wrapAnthropic, or wrapGemini, and turn on captureOptions for input and output messages. The SDK doesn't record message content by default, and Statsig skips spans without messages.
• Call the model inside prompt.withLive() or prompt.withVersion(). Both functions tag each span with the config name and version. A span without a config doesn't show on any config's Traces tab.
• Wrap each multi-step agent turn in one workflow span with wrap({ type: "workflow" }).

The SDK still calls a config a prompt, so you load one with getPrompt(). To send spans without the SDK, set the attributes in the span attribute reference on each span yourself.

The examples on this page share this setup:

import OpenAI from "openai";
import { StatsigAI, initializeTracing, wrapOpenAI } from "@statsig/statsig-ai";

// Sends spans to Statsig with the server secret key in STATSIG_SDK_KEY
initializeTracing({ serviceName: "support-bot" });

const statsigAI = new StatsigAI({ sdkKey: process.env.STATSIG_SDK_KEY });
await statsigAI.initialize();

// Record the input and output messages that online evaluators read
const openai = wrapOpenAI(new OpenAI(), {
  captureOptions: {
    capture_input_messages: true,
    capture_output_messages: true,
  },
});

Grade a single model call

When an agent turn makes one model call, Statsig grades that call. The SDK adds the statsig.span.llm_root marker to each model call that isn't inside another model call or a workflow span. An HTTP span or another app span around the call doesn't change the marker. Statsig grades the spans that carry this marker.

The following example answers with the Live version. It also runs each Candidate version on the same question in the background. Each call produces its own span, tagged with its version, so Statsig grades each version separately.

// Calls the model with one config version
function callModel(version, question) {
  return openai.chat.completions.create({
    model: version.getModel({ fallback: "gpt-5-mini" }),
    messages: version.getPromptMessages({ question }),
  });
}

async function answerQuestion(user, question) {
  const prompt = statsigAI.getPrompt(user, "support_bot");

  // Candidates answer in the background. Users never see these answers.
  for (const candidate of prompt.getCandidates()) {
    prompt
      .withVersion(candidate, (version) => callModel(version, question))
      .catch(console.error);
  }

  const response = await prompt.withLive((version) =>
    callModel(version, question),
  );
  return response.choices[0].message.content;
}

Grade a multi-step agent turn

When one turn makes several model calls or tool calls, wrap the turn in a workflow span. The workflow span ends after every step in the turn, so it can hold the turn's input and final answer. Statsig grades the workflow span as the turn's answer. Statsig never grades tool spans, such as spans from wrap({ type: "tool" }).

The SDK doesn't record messages on workflow spans. Before the workflow function returns, set gen_ai.input.messages and gen_ai.output.messages on the active span to the turn's input and final answer. Without these attributes, Statsig skips the workflow span. The SDK doesn't mark the model calls inside a workflow span, so Statsig then grades nothing in the turn. Start the turn inside withLive() or withVersion() so the workflow span records the config and version.

import { trace } from "@opentelemetry/api";
import { wrap } from "@statsig/statsig-ai";

// One workflow span for each agent turn
const runTurn = wrap(
  { type: "workflow", name: "support_turn" },
  async (version, question) => {
    const model = version.getModel({ fallback: "gpt-5-mini" });

    // Step 1: draft an answer from the config's prompt
    const draft = await openai.chat.completions.create({
      model,
      messages: version.getPromptMessages({ question }),
    });

    // Step 2: revise the draft
    const revision = await openai.chat.completions.create({
      model,
      messages: [
        {
          role: "system",
          content: "Correct any errors in this answer and shorten it.",
        },
        { role: "user", content: draft.choices[0].message.content ?? "" },
      ],
    });
    const answer = revision.choices[0].message.content ?? "";

    // The SDK doesn't record messages on workflow spans, so set the turn's
    // question and final answer for online evaluators to read
    trace.getActiveSpan()?.setAttributes({
      "gen_ai.input.messages": JSON.stringify([
        { role: "user", content: question },
      ]),
      "gen_ai.output.messages": JSON.stringify([
        { role: "assistant", content: answer },
      ]),
    });
    return answer;
  },
);

async function answerWithAgent(user, question) {
  const prompt = statsigAI.getPrompt(user, "support_bot");
  // Start the turn inside withLive so the workflow span records the config and version
  return prompt.withLive((version) => runTurn(version, question));
}

Turn on online grading

Online grading starts when an evaluator has an enabled online rule that matches your spans.

1. Open an evaluator that can run online, or create one. LLM as a Judge and Python evaluators can run online when they read only {{input}}, {{output}}, and {{trajectory}}. For details, go to Evaluator types.
2. On the evaluator's Setup tab, under Targeting rules, click Add rule and check Online.
3. Add conditions that match your spans. For example, choose Config, choose is any of, and select your config.
4. In Sample, enter the percentage of matching traces to grade. New rules start at 10%.
5. Click Save Evaluator. A project or org admin must save any change to an enabled online rule.

For every rule field and operator, go to Online rules.

Online rules choose spans in these ways:

• Each evaluator grades a span under the first enabled online rule that matches the span.
• Statsig samples by trace. Statsig hashes the trace ID, so every span in a trace gets the same sampling decision.
• To stop grading a config's spans, disable the rules that name the config, or remove the config from their conditions. Rules that don't name the config keep grading its spans.
• A rule without a Config condition can also grade spans that don't belong to any config, such as model calls outside withLive() and withVersion(). Scores for these spans appear in the trace drawer and on the evaluator's Traces tab, but not on a config.

What Statsig grades

Statsig checks each span when it arrives. Statsig grades a span when every requirement in this table holds:

Statsig assigns each score to the config version in statsig.llm.prompt_version_id. For spans from older SDKs that record only the version name, Statsig finds the version by its name.

An evaluator reads three values from the span. {{input}} holds the input messages, {{output}} holds the final assistant answer, and {{trajectory}} holds the tool calls and results before that answer. For message formats and every attribute, go to the span attribute reference. For the environment values, go to Environments.

Online grading limits

Online grading has these limits:

• Latency: A score appears on its span about one minute after Statsig receives the span. The Online evaluators chart can take longer to show the score.
• Hourly cap: Statsig grades at most 1,000 spans per hour for each config. A span counts toward its config's cap when every rule that grades it has a "Config is any of" condition that includes the config. All other graded spans, including spans without a config, share one cap of 1,000 spans per hour for your company. Statsig doesn't grade spans over a cap, and the trace drawer shows No evaluator scores this span.
• One grade per span: Each evaluator grades a span at most once. Statsig retries failed judge calls. If an evaluator still fails, the span has no score from that evaluator.
• Included Credits: When your company's Included Credits run out, Statsig keeps running the evaluators that use your own provider API key. Statsig skips the other evaluators. The trace drawer explains that Statsig didn't grade the span. An admin adds API keys in Settings > Product Configuration > General, under AI Provider API Keys. For which evaluators use credits, go to Evaluator costs. For how Included Credits work, go to Credits and API keys.
• Data residency: Statsig processes online evaluation traces in the US only. Statsig plans to support EU data residency.

Review online evaluation results

Statsig shows online grades on the config's Traces tab, on each span, and on each evaluator's page.

Online evaluators chart

To open the chart, go to Agent Experimentation > Configs, open the config, and click the Traces tab. The Online evaluators chart appears above the span list and covers the last day by default. Choose a view:

• Single version: Each evaluator's average score over time, for the version you select or for All versions. Grades that your code logs with logEvalGrade appear as separate lines, labeled with the grade name and "(SDK)".
• Compare: Select the versions to compare with the Live version. For each evaluator, the chart shows the score change relative to the Live version. The legend marks each change as Better, Worse, or No significant change.

How Statsig compares versions

The Compare view tests each evaluator's score change for statistical significance at p < 0.05. The test depends on where the scores come from:

Scores on a span

To open a span's scores, click the span in the list, then click the Evaluators tab in the trace drawer. Each score shows its evaluator and its score. Online grades show the evaluator type, such as LLM as a Judge or Python. Grades from logEvalGrade show SDK. For the full tab, go to Evaluators tab.

An evaluator's Results and Traces tabs also list its online grades. Statsig tracks scores by evaluator ID, so a renamed evaluator keeps its history. For details, go to Review an evaluator's results.

Log grades from your code

Use logEvalGrade to record a score that your own code computes, such as a user's rating or a business outcome. Statsig shows these grades as SDK lines on the Online evaluators chart and as SDK scores in the trace drawer.

statsigAI.logEvalGrade(user, version, 1, "ticket_resolved", {
  sessionId: conversationId,
  traceId,
  spanId,
  reasoning: "The customer closed the ticket without escalating.",
  metadata: { channel: "chat" },
});

// Send the grade events to Statsig
await statsigAI.flushEvents();

logEvalGrade takes these arguments, in order:

The evalData object accepts these fields:

To grade a workflow span after it ends, such as when a user rates an answer later, save the span's IDs during the turn. Inside the workflow function, trace.getActiveSpan()?.spanContext() returns the workflow span's traceId and spanId.

Grade events in your project

Statsig logs each online grade as a statsig::eval_result event in your project. The Online evaluators chart reads these events. The event's value is the config name, or empty for a span without a config. The events have no user ID.

Statsig logs these events with a server secret key that has the description AI_EVALS. Statsig creates the key in your project the first time it logs an online grade.

logEvalGrade logs the same event name with source set to sdk. Its events also carry session_id, is_live, and the fields you pass in metadata. Evaluator metrics in experiments that you create from a config read only grades from logEvalGrade. Online grades don't feed those metrics.