On this page

Set up AI Evals

End-to-end walkthrough for setting up Statsig AI Evals: create a prompt, run offline evals against a dataset, serve prompts in code, and grade production traffic with online evals.

Early Access

This feature is in Early Access. During this time, aspects of the functionality may still be developed, and this documentation may not always be up to date. If you have any questions, contact Statsig Support.

Statsig AI Evals score the output of your prompts, both before and after you ship them. This walkthrough covers the full path in the Statsig console and in your code: create a prompt, score it against a dataset, serve it from your app, and grade real traffic.

AI Evals give you two feedback loops:

  • Offline evals: Grade a prompt against a fixed dataset before you ship, so you catch regressions early.
  • Online evals: Grade real production output after you ship, including shadow-running candidate versions your users never see.

Create a prompt

1. Create the prompt

Go to AI Evals > Prompts and select Create New Prompt. Give it a Name (this is the name your code references), an optional Description, and a Target Application. You can also set a Team and Tags.

2. Configure the version

Open the prompt. On the Versions tab, Prompt Setup subtab, configure:

  • Provider and Model: The available models come from your project's configuration, so the list reflects what your project supports.
  • Parameters: Set Temperature and Max tokens, plus optional Top P, Frequency Penalty, and Presence Penalty. Some reasoning models use a fixed temperature and disable the control.
  • Output Format: Choose Text, JSON Object, or Structured Output (JSON Schema).
  • Messages: Add System, User, and Assistant messages. Insert variables with double-brace macros like {{input}}. A macro name matches a column in your eval dataset, or a value you pass at runtime. Statsig reserves {{output}} for the model's response.

3. Understand versions

A prompt has four version types. New versions start as Draft:

TypeServed to your SDK?Shown to users?Use it for
DraftNoNoIterating in the console before you serve a version.
CandidateYesNoShadow-running and grading a version without exposing it.
LiveYesYesThe version Statsig actively serves to users.
ArchiveNoNoInactive versions you keep for reference.

Your SDK only receives Live and Candidate versions. Promote versions from the version menu with Promote to Live, Set to Candidate, Set to Draft, or Archive.

Run an offline eval

Offline evals score a prompt version against a fixed dataset. Configure a dataset and graders on the prompt's Eval Setup tab, then run the eval from the Versions tab.

1. Add a dataset

On the Eval Setup tab, under Dataset, select Select a Dataset, then either Upload Dataset (a CSV) or browse an existing one. The first CSV row holds the column names, and you define the columns yourself, for example an input column and a reference_output column. A categories column (comma-separated) breaks results down by segment. You can also build a dataset by hand with Add Row and Add Column on the Datasets page.

2. Add graders

Under Graders, select Add New Grader and pick a type:

  • LLM as a Judge: Another model scores the output against a rubric you write. Reference the model output with {{output}}.
  • String Comparison: Compare {{output}} to a column with Equals, Does not equal, Contains, or Contains (ignore case).
  • Text Similarity: Score how close {{output}} is to a column using a function like fuzzy match or cosine similarity.
  • Python: Score the output with your own Python.

Every grader returns a score between 0 and 1. Mark one grader Primary to drive the version's overall score. Mark a grader Critical when it's a must-pass check: if a critical grader scores 0, Statsig fails the whole run.

3. Run the eval

Go to the Versions tab, Prompt Setup subtab, and select Start Evaluation. Select the versions you want to score and confirm with Run Eval. The button stays disabled until the prompt has both a dataset and at least one grader.

4. Read the results

Results appear on the Eval Results subtab: an overall Score gauge, a card per grader, a breakdown by category, and a per-row table. Select any row to inspect its input, the model's output, and the score. Use Compare to diff versions, and Promote to Live when a version is ready.

Iterate by creating new versions of the prompt, running the eval on each, and comparing scores. Promote the best version to Live to serve it.

Serve the prompt in your app

Retrieve the prompt at runtime with the Statsig AI SDK and run a completion with your provider. First, install and initialize the SDK.
npm install @statsig/statsig-ai

import { StatsigAI } from '@statsig/statsig-ai';

const statsigAI = new StatsigAI({ sdkKey: 'YOUR_SERVER_SECRET_KEY' });
await statsigAI.initialize();

Fetch the prompt and use its Live version. Statsig serves the prompt config; your app makes the completion call with your own provider client (OpenAI here, but any provider works). getPromptMessages fills your {{macros}} and returns messages ready to pass to the provider.

import { StatsigUser } from '@statsig/statsig-ai';

const user = new StatsigUser({ userID: 'a-user' });

const prompt = statsigAI.getPrompt(user, 'support_summary');
const live = prompt.getLive();

const response = await openai.chat.completions.create({
  model: live.getModel({ fallback: 'gpt-4.1' }),
  temperature: live.getTemperature(),
  max_tokens: live.getMaxTokens(),
  messages: live.getPromptMessages({ input: userInput }),
});

Grade production traffic with online evals

Online evals grade real production output, without a ground-truth answer to compare against. You can also shadow-run Candidate versions in the background, so you gather grades on a new version before any user sees it.

Serve the live and candidate versions

Show the Live output to your user, and run Candidate versions in the background without showing them. Statsig serves each version's config; your app makes the completion calls, so your code runs both the live and the candidate completions.

js
const prompt = statsigAI.getPrompt(user, 'support_summary');
const live = prompt.getLive();
const candidates = prompt.getCandidates();

// Show the live output to the user
const liveResponse = await openai.chat.completions.create({
  model: live.getModel({ fallback: 'gpt-4.1' }),
  temperature: live.getTemperature(),
  max_tokens: live.getMaxTokens(),
  messages: live.getPromptMessages({ input: userInput }),
});

// Shadow-run each candidate in the background. Don't show these to the user.
for (const candidate of candidates) {
  const candidateResponse = await openai.chat.completions.create({
    model: candidate.getModel({ fallback: 'gpt-4.1' }),
    temperature: candidate.getTemperature(),
    max_tokens: candidate.getMaxTokens(),
    messages: candidate.getPromptMessages({ input: userInput }),
  });
  // Grade candidateResponse and log the grade (covered below)
}

Choose how to produce grades

There are two ways to turn production output into grades. You can use either or both.

Automatic grading lets Statsig grade a sample of your live traffic for you:

  1. On the prompt's Eval Setup tab, turn on Enable for online eval for a grader.
  2. Under Online Configuration, set a Sampling Rate, the percentage of traffic Statsig grades.
Statsig grades the sampled traffic from the OpenTelemetry traces your application sends, so you need telemetry flowing. The Node AI SDK provides an initializeTracing function and a wrapOpenAI helper to emit these traces (Python tracing support is coming soon). Online graders have no ground-truth column to compare against, so they grade from the model output alone. Use an LLM-as-a-judge grader whose only variable is {{output}}, or a Python grader that doesn't read dataset columns.

Manual grading lets you grade the output in your own code and log the score. Scores must be between 0 and 1.

statsigAI.logEvalGrade(user, live, 0.92, 'helpfulness', {
  sessionId: sessionId,
});

// Flush eval grade events to Statsig
await statsigAI.flushEvents();

View online results

Open the prompt, go to the Versions tab, Eval Results subtab, and switch to Online Results. Each version's exposures and grader scores appear over time. Compare Candidate versions against Live for grader deltas relative to the Live baseline, before you promote a candidate.

Next steps

Was this helpful?