SEO Experimentation with Statsig

Learn how to run SEO experiments to test landing page designs and optimize organic traffic performance.

SEO experimentation tests changes to your web pages, such as titles, metadata, structured data, or layout, to measure how they affect search rankings and organic traffic. Because search crawlers need a consistent version of each URL, you randomize by page instead of by user. Statsig hashes each canonical URL into a control or test bucket, then compares metrics such as impressions, average position, and organic sessions across the two groups. Use it when you run a large, templated site and want proof that a page change moves organic traffic before you roll it out everywhere. If your change affects a shared system over time rather than individual pages, run a switchback test instead.

Airbnb's growth team developed this approach in 2017 to test a new landing page template across more than 100,000 city URLs, where user-level randomization was impossible because crawlers needed consistent page versions.

Select a deterministic page bucket

Crawlers must see a consistent version of each URL during the test window, so you can't randomize by user. Instead, hash the canonical URL into buckets. In Statsig, add page_url as a custom unit ID:

From Project Settings, go to Custom Unit IDs.
Enter a name and description. The custom unit ID is immediately available to experiments, gates, and dynamic configs.

Statsig then deterministically hashes each page into control or test and keeps the assignment stable across sessions.

Strip the http or https scheme and any query parameters from page_url, so that Statsig hashes only the stable base URL.

Define metrics before shipping

Make sure the metrics you want to measure are in your data warehouse, keyed on page_url. Register them in Statsig's metric catalog. Because the same pipeline powers feature experiments, your existing CUPED or stratified-sampling settings apply automatically.

Example metrics

Layer	Metric source	Why it matters
Indexing lag	Impressions, average position (Search Console)	Early signal during re-crawling
Primary KPI	Organic sessions keyed by `page_url` (Statsig events)	Measures traffic that lands
Quality guardrail	Conversion, bounce, read-depth, revenue	Confirms the traffic is useful

Implement the change behind an experiment

Create an experiment in the Statsig console, for example seo_title_test.
Target the custom unit ID page_url with a 50/50 split across control and test.
Expose the variant in the template renderer or CDN edge function.

Ship, monitor, and decide

Use power analysis to determine how long to run the experiment based on traffic volume.
Expect first signals in 2 to 7 days. Wait for re-indexing to plateau before you treat results as stable.
Merge the winner into your template and archive the test. Experiment summaries remain searchable after archival.

SEO-specific guardrails

Guardrail	What to watch	Why it protects you
Indexation change	`indexed_pages` compared with baseline	A template change that blocks crawling (robots, canonicals, noindex) shows a sharp drop long before traffic falls
Cannibalization ratio	Average URLs served per query	Multiple pages newly ranking for the same query dilute CTR and can reduce combined traffic
HTTP response mix	Percentage of 410, 301, and 200 responses	A bulk 410 (gone) or misconfigured 301 can remove long-tail pages from the index
Core Web Vitals drift	LCP and CLS at p75	Page-speed regressions can hurt rankings without other symptoms
Crawl budget	Average TTFB and bytes per page	Slow or bloated pages decrease crawl rate

Page-level changes worth testing

Theme	Why it might move organic traffic	Typical implementation
Title and meta variants	Query matching, CTR uplift	Add or remove a brand suffix, switch noun phrasing to verb phrasing, insert a dynamic price
Structured data	Rich-result eligibility	Inject FAQ, HowTo, Breadcrumb, or Product schema blocks
Internal-link blocks	Crawl priority and PageRank flow	Reorder the related-articles widget; test link count caps
Content snippets	Relevance and long-tail keywords	Auto-generate a 50-word intro compared with none; expand FAQ length
Canonical and hreflang tags	Duplicate-content handling	Toggle self-canonical compared with cluster canonical; add `hreflang="x-default"`
Media handling	CLS and LCP scores influence rankings	Defer off-screen images; inline the critical hero image; switch to AVIF
Pagination model	Crawl depth and index coverage	Compare classic `?page=` URLs, `rel="next/prev"`, and load-more buttons
Performance budgets	Core Web Vitals ranking factor	Split JS into 200 ms chunks compared with baseline; purge CSS and inline critical CSS
Ad layout	CLS penalties, user engagement	Reserve fixed ad slots compared with dynamic slots; lazy-load below the first viewport
Schema position	Parser friendliness	Move the JSON-LD block to `<head>` compared with the end of `<body>`

Was this helpful?