Forward Proxy

Statsig Forward Proxy

The Statsig Forward Proxy is a service you can host in your own infrastructure. When you configure SDKs to use the forward proxy, the proxy provides a closer and more reliable hop for retrieving Statsig configurations from within your infrastructure instead of reaching out to Statsig's networks.

The expected benefits of using the proxy include:

1. Reduced dependency on Statsig Infrastructure Availability
2. Improved performance through localization of download_config_spec to your cluster
3. Improved cost savings and reduced network overhead through minimizing requests to Statsig Infrastructure
4. Improved consistency of configurations, as the forward proxy becomes an articulation point for serving configs
5. Support for cool features such as a streaming API through GRPC

If you have questions about deployment or need assistance, contact Statsig on Slack or create a new GitHub issue.

Installation

Helm Installation

The recommended way to install Statsig Forward Proxy in a Kubernetes environment is the official Helm chart. Follow these steps to deploy:

# Add the Statsig Helm repository
helm repo add statsig https://statsig-helm.storage.googleapis.com
helm repo update

# Install the chart
helm install statsig-forward-proxy statsig/statsig-forward-proxy

Manual Deployment

For environments where Helm isn't available or for more customized deployments, you can deploy Statsig Forward Proxy manually. The proxy is available as a pre-built Docker image, and you can also build your own binary from source.

How Forward Proxy works

The forward proxy works by setting up a local HTTP or gRPC server. When you make requests to the forward proxy, only the initial request is blocking. After that, a background loop keeps configurations up to date without impacting the serving path.

The forward proxy also enables backup caches and monitoring through technologies such as Redis and Statsd.

Integration with SDK

You can configure SDK networking to use different protocols for integration with the Statsig Forward Proxy. The SDK makes requests to three network endpoints: download_config_specs, get_id_lists, and log_events. You can configure download_config_specs to use the proxy. Support for the latter two is under active development.

Network protocol for different endpoints

For download_config_specs endpoint, where we get specs on evaluating gates/layers/experiments/configs

1. http: the default protocol. If the SDK is initialized with http, it polls config updates in a background thread.
2. grpc_websocket: Establishes a gRPC streaming connection from the SDK to the Statsig Forward Proxy and listens for updates pushed from proxy servers. You must use the Statsig Forward Proxy or a similar proxy server to use gRPC streaming. Not all SDKs support gRPC yet. If you need support for a specific SDK, reach out on the Statsig support channel on Slack.
3. grpc: Unary RPC, behaves similarly to the HTTP protocol. After initialization, the SDK polls changes from the forward proxy server in a background thread.

Listen to Config Change Example

const statsigOptions = {
  proxyConfigs: {
    download_config_specs:  {
      proxyAddress: proxyAddress // Your proxy address
      protocol: "grpc_websocket"
    }
  }
}
Statsig.initialize(server_key, statsigOptions)
// Statsig will use listen for config updates from statsig forward proxy using grpc_websocket protocol. And use http to get idlists and post log events from statsig servers.

For information on your specific SDK language, go to the language-specific docs in the left-hand column.

Failover behavior

Initialization

SDK behavior remains the same when the forward proxy is in use. Several configurations can improve data availability during initialization. The default behavior is to get from the forward proxy. If a DataAdapter is present, the SDK gets from the DataAdapter first and falls back to the Forward Proxy if no value is returned from the data adapter. Several ways exist to configure this behavior.

1. Set fallbackToStatsigAPI = true to fall back to the Statsig API when initialization or config sync from primary sources fails.
2. To customize the initialization order, set initializeSources with multiple sources. For example, initializeSources=[DataSource.Network, DataSource.DataAdapter, DataSource.StatsigNetwork] fetches from three sources sequentially until one succeeds.

Config Sync (Post initialization)

If the SDK is using a polling model (grpc or http):

1. Set fallbackToStatsigAPI=true.
2. Set configSyncSources with multiple sources. For example, initializeSources=[DataSource.Network, DataSource.StatsigNetwork] fetches from sources sequentially until one succeeds. Config sync sources don't apply when you are using streaming.

If the SDK is using the grpc_websocket protocol (listening mode for config changes), by default it will:

1. Retry connecting to the forward proxy with exponential backoff: 10s, 50s, 250s, 1250s, and so on, up to 10 retries.
2. If the 4th retry fails, the SDK starts polling from the Statsig endpoint. Use StatsigOptions.pollingInterval to control the polling frequency.
3. While polling from the Statsig endpoint, the SDK continues retrying the forward proxy connection until it succeeds.

All of the above behaviors are configurable through StatsigOptions.

Streaming failover in depth

When exceptions occur (for example, when a connection is dropped or the forward proxy is down), gRPC returns an error to the SDK, which then starts the failover behavior. The Python SDK checks whether the channel is idle every 2 hours to prevent edge-case behavior in the Python gRPC library.

TLS: advanced network authentication

The forward proxy supports TLS and mTLS when using the gRPC server, so all gRPC network traffic (streaming and unary calls) is encrypted.

To enable it:

1. Setup forward proxy server with valid certifications
2. Setup SDK with valid certifications (python example).

Coordinate SDK DataAdapter / DataStore with Forward Proxy cache service

To increase reliability during initialization, configure a DataStore. If the Forward Proxy is unavailable at initialization time, the SDK initializes with values from the DataStore and then syncs from the forward proxy in the background. The SDK and the Forward Proxy should share the same cache service for consistency and to reduce cache I/O.

Example on DataStore setup in python:

class DataAdapter(IDataStore):
def __init__(self):
  self.redis_cache = ExampleCache()

def get(self, key: str):
  # IDlist isn't currently supported by proxy, so do normal lookup
  if "statsig.id_lists" in key:
    return self.cache.get(key)
  # This logic must stay in sync with statsig-forward-proxy
  else:
    hashed_key = "statsig::" + hashlib.sha256(key.encode()).hexdigest()
    return self.cache.hget(hashed_key, 'config')

def set(self, key: str):
  # Don't implement set method if you are share the same cache between forward proxy and sdk, forward proxy will write to cache 
  pass

def shutdown(self):
  self.cache.shutdown()

Deployment Notes

Because every use case differs, there are no strict requirements. General recommendations include:

1. Understand your general QPS to Statsig. Contact us if you need help.
2. Estimate the throughput a single pod can support, then calculate how much to scale out to handle your traffic.
3. Pre-scale the proxy for your peak QPS before gradually rolling it out to services.
4. Configure auto-scaling. For most deployments, scaling up at 70% CPU and memory utilization is sufficient, but this may need adjustment depending on usage.

If horizontal scaling doesn't resolve scaling issues, front the proxy with nginx to help with flow control.