---
title: Offline Evals
description: Run offline AI evaluations in Statsig to grade model outputs against fixed test sets and catch regressions before exposing changes to real users.
product: general
token_estimate: 998
---
# Offline Evals

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

## What are offline evals

Offline evals provide quick, automated grading of model outputs on a fixed test set. They catch wins and regressions before you expose changes to any real users. For example, compare a new support bot's replies to human-curated answers to decide if the bot is ready to ship. Use offline evals when you have a fixed test set with ideal answers to grade against. Use online evals instead when you want to grade output on live production traffic without a ground truth to compare against.

Steps to run offline evals on Statsig:

1. Create a Prompt that contains the instruction for your task (for example, "Classify tickets as high, medium, or low urgency based on ticket text").
2. Upload a sample dataset with example inputs and ideal answers (for example, Ticket1 text to High; Ticket2 text to Low).
3. Run your AI on the dataset to produce output (for example, classify each ticket).
4. Grade or score the outputs by comparing the ideal answers in the dataset with the AI-generated output.
5. Create multiple versions of your prompts, compare scores across versions, and promote the best one to Live.

> **Info:**
>
> For a start-to-finish walkthrough that also covers serving prompts in code and running online evals, refer to [Set up AI Experimentation](https://docs.statsig.com/ai-evals/setup).

## Create/analyze an offline eval in 10 minutes

**1. Create a Prompt within Statsig**

This captures the instruction you provide to an LLM to accomplish your task. Use the Statsig [Node](https://docs.statsig.com/ai-evals/node#getting-a-prompt) or [Python](https://docs.statsig.com/ai-evals/python#getting-a-prompt) AI SDKs to retrieve this prompt within your app. You can create multiple versions of the prompt as you iterate and choose which one is "live" (the version the SDK retrieves).

![Statsig prompt editor listing live and candidate versions with messages](https://docs.statsig.com/images/ai-evals/offline-evals/a17b3c4d-2126-4dfe-8d4b-d40b1838f878.png)

**2. Create a dataset you can use to evaluate LLM completions for your prompt**

For the example above, this might be a list of words alongside known good translations in French. You can enter small lists manually, or upload a CSV.

![Dataset creation table with translation pairs for offline evaluation](https://docs.statsig.com/images/ai-evals/offline-evals/6d4b1abc-bde9-4d63-9d0c-95fef60b3f9a.png)

**3. Create a grader that grades LLM completions for your prompt**

Configure a grader that scores each completion. You can compare the completion to a reference column with a **String Comparison** or **Text Similarity** grader, or configure an **LLM as a Judge** grader that mimics a human's grading rubric. Every grader returns a score between 0 and 1.

![Grader configuration form comparing model output against reference answers](https://docs.statsig.com/images/ai-evals/offline-evals/3cd510f7-c267-4cdd-bebe-dbee527a5318.png)

**4. Run evaluation**

Run an evaluation on a version of the prompt. Results appear in a few minutes and look like this. You can click into any row of the dataset to understand more about the evaluation for that row.

![Offline evaluation results table showing prompt version scores](https://docs.statsig.com/images/ai-evals/offline-evals/c450f277-b2ba-4657-b747-440b43859f20.png)

You can categorize your dataset, and break scores out by category.

![Category breakdown chart splitting evaluation scores by dataset segments](https://docs.statsig.com/images/ai-evals/offline-evals/3c0de7c4-6721-4a45-9a61-04a63db68913.png)

If you have scores for multiple versions, you can compare them to see what changed between versions.

![Comparison view charting multiple prompt versions across graders](https://docs.statsig.com/images/ai-evals/offline-evals/fd593e52-ddec-4826-bf4b-c2ca1d43e4f0.png)