Deploy Forward Proxy ID list enrichment

Statsig Forward Proxy synchronizes ID lists into Redis and serves membership lookups through POST /v1/user_enrichment. Use this deployment to share ID list data across application processes instead of downloading every list into each SDK.

This guide covers proxy deployment and operation. After the endpoint is ready, follow SDK user enrichment to build enriched users and evaluate them locally. For config-spec routing alone, use the regular Forward Proxy setup.

How synchronization works

All Forward Proxy pods participate in downloading ID lists and updating shared Redis data. The proxy periodically checks for changes, downloads new or updated list data, and removes lists that are no longer needed.

Server keys for the same company share the stored ID list data. Each key's enrichment responses include only the lists available to that key.

Configure the same SFP_IDLIST_SYNC_KEYS_JSON key set on all pods that share a REDIS_KEY_PREFIX so they synchronize the same lists.

Supported ID list sizes

The standard project limit is 10 million IDs across all ID lists. To exceed this limit, contact Statsig Support for enablement. This requirement applies to both standard SDK downloads and Forward Proxy enrichment. Deploying Forward Proxy doesn't raise the limit automatically.

With standard SDK downloads, each process needs memory for its lists. With enrichment, Redis stores the shared lists and membership indexes. Size Redis for the enabled workload before deployment.

Prerequisites

• A Statsig server secret key for the project that owns the ID lists.
• An enrichment-capable Forward Proxy image supplied for early access. Pin its tag or digest. Contact Statsig Support for a compatible build.
• Helm 3 for Kubernetes, or Docker with Docker Compose for local use.
• Redis that all Forward Proxy replicas can reach. Use persistence and noeviction for production because evicting reverse-index keys can produce incomplete memberships.

Start Forward Proxy and Redis with Docker Compose

Create .env next to your Compose file:

STATSIG_SERVER_SDK_KEY=secret-your-server-key
STATSIG_FORWARD_PROXY_IMAGE=statsig/statsig-forward-proxy:<enrichment-image-tag>

Keep .env out of version control. In production, inject STATSIG_SERVER_SDK_KEY from your secret manager.

Create compose.yaml:

services:
  redis:
    image: redis:7-alpine
    command:
      - redis-server
      - --appendonly
      - "yes"
      - --maxmemory-policy
      - noeviction
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 20
    restart: unless-stopped

  forward-proxy:
    image: ${STATSIG_FORWARD_PROXY_IMAGE:?Set an enrichment-capable image}
    depends_on:
      redis:
        condition: service_healthy
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      REDIS_ENTERPRISE_HOST: redis
      REDIS_ENTERPRISE_PORT: "6379"
      REDIS_TLS: "false"
      REDIS_CONNECTION_POOL_MAX_SIZE: "64"
      SFP_IDLIST_SYNC_KEYS_JSON: '["${STATSIG_SERVER_SDK_KEY}"]'
      IDLIST_SYNC_INTERVAL_IN_S: "60"
      IDLIST_REDIS_POOL_SIZE: "32"
      IDLIST_FETCH_TIMEOUT_MS: "50"
      USER_ENRICHMENT_MAX_LOOKUPS: "100"
    command:
      - http
      - redis
      - --http-request-timeout-in-s
      - "1800"
      - --http-read-timeout-in-s
      - "60"
    shm_size: "1gb"
    restart: unless-stopped

volumes:
  redis-data:

Replace <enrichment-image-tag> with the compatible tag before starting the services. Setting SFP_IDLIST_SYNC_KEYS_JSON enables synchronization and the enrichment endpoint together.

Start the services:

docker compose up -d
docker compose logs -f forward-proxy

--http-request-timeout-in-s 1800 gives large ID list downloads time to stream. Size this value for your largest list and available network throughput. The default 30-second request timeout is often too short for large lists.

Start Forward Proxy with Helm

Use the official Helm chart for Kubernetes with an enrichment-capable image, Redis cache mode, and SFP_IDLIST_SYNC_KEYS_JSON.

helm repo add statsig https://statsig-helm.storage.googleapis.com
helm repo update

Store authorized server keys in a Kubernetes Secret:

kubectl create secret generic sfp-idlist-sync-keys \
  --from-literal=SFP_IDLIST_SYNC_KEYS_JSON='["secret-YOUR_SERVER_SDK_KEY"]'

Create values.yaml:

replicaCount: 2
image:
  repository: statsig/statsig-forward-proxy
  tag: "<enrichment-image-tag>"
sfp:
  args:
    - http
    - redis
    - --http-request-timeout-in-s
    - "1800"
    - --http-read-timeout-in-s
    - "60"
  environment:
    REDIS_ENTERPRISE_HOST: redis.example.internal
    REDIS_ENTERPRISE_PORT: "6379"
    REDIS_TLS: "false"
    REDIS_CONNECTION_POOL_MAX_SIZE: "64"
    IDLIST_SYNC_INTERVAL_IN_S: "60"
    IDLIST_REDIS_POOL_SIZE: "32"
    IDLIST_FETCH_TIMEOUT_MS: "50"
    USER_ENRICHMENT_MAX_LOOKUPS: "100"
  secrets:
    - envName: SFP_IDLIST_SYNC_KEYS_JSON
      secretName: sfp-idlist-sync-keys
      secretKey: SFP_IDLIST_SYNC_KEYS_JSON

Install the chart:

helm install statsig-forward-proxy statsig/statsig-forward-proxy -f values.yaml

Point REDIS_ENTERPRISE_HOST at a Redis deployment that every replica can reach. Use persistence and maxmemory-policy noeviction. For additional Helm options, refer to the Forward Proxy Helm chart documentation.

Verify the deployment

Check that the proxy responds:

curl --fail http://localhost:8000/v1/health
curl --fail http://localhost:8000/v1/ready

These checks report HTTP service health. They don't verify Redis connectivity or completion of ID list synchronization.

Export the same server key configured in SFP_IDLIST_SYNC_KEYS_JSON, then request memberships for a known test user:

export STATSIG_SERVER_SDK_KEY=secret-your-server-key
curl --fail-with-body http://localhost:8000/v1/user_enrichment \
  -H "statsig-api-key: ${STATSIG_SERVER_SDK_KEY}" \
  -H 'Content-Type: application/json' \
  --data '{"user":{"userID":"user-123","customIDs":{"accountID":"account-456"}}}'

The endpoint accepts raw IDs in user for this check and hashes them internally. The SDK sends prehashed values in id_list_lookups. A successful response contains id_lists and unsupported_enrichments. The id_lists object maps each lookup hash to an array of matching list names.

Use a known member and confirm the expected list name. An empty lookup request skips Redis, so it can't test index readiness.

ID list updates aren't available immediately. Until changes finish syncing, membership-based targeting can use stale data. A newly added user may appear as a non-member, while a removed user may still appear as a member. HTTP 200 confirms a successful lookup, but doesn't guarantee that the result reflects the latest list changes. Allow time for synchronization and verify expected membership results before relying on updated targeting.

Forward Proxy configuration

The background poller refreshes manifests through the normal proxy polling loop. Set this interval with --polling-interval-in-s (default 10). IDLIST_SYNC_INTERVAL_IN_S controls producer scheduling separately. The old --idlist-sync-enabled, --user-enrichment-enabled, and IDLIST_MANIFEST_POLL_INTERVAL_IN_S settings don't apply to this implementation.

Required settings

Enrichment request settings

Synchronization settings

Normal version changes use partitioned streaming diff for every list size. Lists below the partition target use one partition. First loads and transitions from an empty list apply new memberships directly without building a difference set. The removed IDLIST_STREAMED_DIFF_MIN_MEMBERS setting has no effect.

Redis connection settings

Use TLS and authentication outside a local development network. For OAuth and certificate requirements, refer to the Forward Proxy Redis authentication guide.

Estimate Redis capacity

ID lists can outgrow a single Redis deployment. Budget for both the forward sets that store each list's members and the reverse mappings that identify which lists contain each user ID.

For initial planning, allow approximately 50–80 bytes per forward-set member and 80–120 bytes per distinct ID for reverse mappings. These are conservative estimates for eight-character lookup hashes, short Redis key prefixes, and compact integer list tokens. They assume one list, one current version, and no shared IDs with other lists. Redis version, allocator, encoding, and key lengths affect actual memory.

The Redis subtotal excludes old and in-progress list versions, other proxy cache data, fragmentation, client buffers, persistence overhead, and replicas. Updates can retain an old forward set alongside the new one and temporarily increase reverse mappings. Use IDLIST_OLD_VERSION_TTL_IN_S to configure how long Redis retains an old forward set after a replacement completes. Allow for every version retained during your update cadence, not just one replacement. A replica stores another copy of the dataset; persistence and replication require additional capacity.

For 100 million IDs, one retained forward version adds 5–8 GB to the 13–20 GB subtotal. That totals 18–28 GB before other overhead and replicas.

Across multiple lists, forward storage grows with total list memberships. Reverse mappings share one key per distinct ID within a company and add list tokens for overlapping memberships. Use your actual overlap and membership counts to size storage across multiple lists.

Load representative data into your Redis version and use MEMORY USAGE and INFO memory to validate the estimate. Monitor resident memory as well as dataset memory, then exercise list replacements and persistence operations. Redis documents additional memory requirements during persistence.

Scale beyond one deployment

• Increase Redis memory capacity before the dataset and update overhead approach its limit. Keep persistence and noeviction enabled.
• Separate independent projects or workloads into dedicated Forward Proxy and Redis deployments. Route each application's enrichment requests to the deployment that holds its lists.
• Add Forward Proxy pods for request and download capacity. More pods don't increase Redis storage capacity. Read replicas can help read throughput, but each stores a full copy of the data.
• If one workload or list exceeds a practical Redis node's memory, contact Statsig Support to plan a supported deployment topology. Confirm the deployment plan before increasing the enabled size.

Production operation

Run multiple proxy replicas against the same Redis deployment. Keep server keys and Redis credentials in a secret manager. Use TLS and authentication outside local development.

Use Redis persistence and maxmemory-policy noeviction. Include the capacity and update overhead described in Estimate Redis capacity. Readiness checks detect missing initialization and dictionary state, but don't detect every missing reverse-index key; eviction can still cause incomplete memberships.

If you use IDLIST_REDIS_READ_URI, account for replication lag before relying on fresh memberships. Benchmark your list sizes, unique IDs, request volume, Redis latency, and replica count instead of assuming a fixed throughput or memory ratio.

Monitor queue depth, task failures, checkpoint lag, staleness, full reloads, delta application, and enrichment 503 responses. Tune the request and read timeouts for your network and largest list.