Skip to main content

Run an A/B Test with Python Flask

Python Flask, a popular lightweight framework, is not only user-friendly but also robust enough for consumer-facing applications. One of the exciting capabilities it offers is the ability to conduct A/B tests efficiently. In this guide, we'll demonstrate how you can A/B test with Python Flask using Statsig's free tier, an essential tool for dynamic web application development.

Today we'll use our Python Server SDK - which works well with Flask out-of-the-box given Flask's limited abstraction on top of Python. Our goal today is to create a website path or "route" in Flask that returns different results based on who the user_id - effectively creating an A/B test.

Step 1: Initial Setup for A/B Testing with Flask and Statsig​

To start A/B testing with Python Flask, first set up your working environment:

mkdir Flask-Flags
cd Flask-Flags
touch app.py

app.py is the file we'll write all of our code in. Next, we should install the Flask and Statsig Python packages if you haven't already. Note, Statsig requires you to use Python 3; for more information, take a look at our Python SDK page.

pip3 install Statsig, Flask

Step 2: Flask Application Basics​

Setting up a basic Flask application is your next step toward implementing A/B testing.

Next, we'll start editing app.py - we'd recommend opening it up in your favorite code editor to get started. If you're new at this, you can open it in any text editor. We'll start with the simplest possible, 6-line flask app. 

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Hello World!"

app.run(debug=True)

You can run this file in your IDE, or on your command line with:

Python3 app.py

If you navigate to http://127.0.0.1:5000 in your browser, you should see "Hello World!".

Next, let's add a new endpoint to our Flask app that we can use for our experiment. We'll call it "experimentRoute". Add this code below your last route ("Hello World"!) but above the app.run call:

@app.route('/experimentRoute')
def experimentRoute():
    return "<body style='background-color:red'/>"

After stopping and starting your project again, you'll be able to see this route at http://127.0.0.1:5000/experimentRoute. You won't see much now, just a red screen!

image

What we'd like to do is make our app show a different color for each user using an A/B test (what we call an "experiment") in Statsig. If our app had more functionality, we could then see which color users preferred or which supported our business metrics and make a decision to choose that option. Even without building more functionality, Statsig could show you differences in metrics like WAU and DAU with a single line of code!

Lets set up an Experiment in Statsig!

Step 3: Implementing A/B Testing in Flask​

If you haven't yet, now is the time to head to console.statsig.com and set up an account. Statsig is free up to 1 Million events per month for A/B Tests.

When you sign up, you'll have the opportunity to set up your SDK type, but if you don't see this, no problem; simply click on the settings gear in your home page, then click on Keys & Environments, where you can copy your server key. Initializing Statsig in Python and Flask only takes two lines:

from statsig import statsig, StatsigUser
statsig.initialize("server-secret-key") #insert your server key here!

Step 4: Setting Up Your First A/B Test with Statsig and Flask​

Next, we'll set up an experiment! Click on experiments in the left-hand side of the Statsig console (or if you're in the onboarding flow, click on "Run an Experiment" and it'll walk you through the whole process). Click on create new and give it a name, I chose color-ab. For now, we can leave the default of user_id.

Next, we should add a hypothesis for our experiment—as is best practice. For this example, we can say "blue will perform better than red." You should also add at least one primary metric to the experiment, for now, we can just chose one like DAU. Then, we need to add parameters to our experiment, which will actually result in different experiences for the user. I've added a parameter called "background-color" with the variants "red" and "blue".

image

Notice that we've set the percentage to 50%, meaning that half of our users (as randomized by the user_id) will see each treatment. Click on save in the bottom right-hand corner, then "start" in the top right, and your A/B test is ready to go.

Let's get it set up in our code. Change your experiment route to look like this:

@app.route('/experimentRoute')
def experimentRoute():
    user = StatsigUser(user_id=f"12345")
    background_color = statsig.get_experiment(user, "color-ab").get("background-color", "green")
    return f"<body style='background-color: {background_color}'>"

Now, when you click on this route: 127.0.0.1:5000/experimentRoute you should be served either a blue or red background, based on which group you've been allocated to in the experiment! Further, as we set our experiment to be randomized on user_id, changing this may result in us being distributed into another group. Lets add user_id as a parameter in our Flask route, so that we can change it on the fly:

@app.route('/experimentRoute')
def experimentRoute():
    userID = request.args.get('userID') #also remember to import the request object from the Flask module!
    user = StatsigUser(user_id=userID)
    background_color = statsig.get_experiment(user, "color-ab").get("background-color", "green")
    return f"<body style='background-color: {background_color}'>"

Now, you are able to set a userID in your URL parameters like such: 127.0.0.1:5000/experimentRoute?userID=1234 and when you vary the parameter - you should see different results!

img

img

Note: You won't see a different one every time due to how our randomization works, but over time, the distribution should be even. If not, you may have a [Sample Ratio Mismatch]((https://www.statsig.com/blog/sample-ratio-mismatch).

Thanks for reading this tutorial! If you have trouble or any other questions, please join our Slack Group.