Bot Traffic

Learn how Statsig filters bot traffic from experiments to ensure clean analytics data.

Statsig removes exposures from known bots, such as search crawlers, AI scrapers, and link previewers, before it computes exposure counts and Pulse results, so metrics reflect real users. Bots still receive gate and experiment values from the SDK; filtering affects analysis only. Follow the segment procedure on this page when you also need to control what bots see, for example to stop a search engine from indexing an unreleased homepage variant.

How bot filtering works

Statsig filters known bots out of exposures data for feature gates, holdouts, layers, and experiments, so the exposure counts and analytics you see reflect real users only. Without filtering, bot traffic from search engines and AI scrapers makes it harder to measure how many real users see your changes. For background, refer to the Statsig guide to online bot filtering.

After Statsig filters bot data from your exposures data, the bot data isn't viewable in the Statsig console. Contact Statsig in Slack if you have questions about filtered data.

Control what bots see in gates and experiments

Statsig doesn't block bots from receiving your feature gates and experiments. The API and SDK serve bots configs and variants following your setup; Statsig filters only their exposures from analysis data and from the exposure count in Pulse.

To restrict what bots see, for example to keep search engines from indexing a new homepage variant before release, create a segment of known bots and apply it to your gates and experiments.

Create a Known Bots segment

Create a new segment named "Known Bots" for your project. The segment must be a conditional segment.

New conditional segment setup for capturing known bots

After you create the segment, add a rule. Set Criteria to Browser Name and leave Operator as Any Of. In the Values field, paste the full string from the Bot list section (use the copy button to the right of the list). The Statsig console splits the pasted list into individual bot names.

Segment rule showing browser name filter populated with bot list

You can reuse this segment for all your launches.

Apply the segment to gates and experiments

For gates, create a new rule that controls the bot experience.

Feature gate rule ensuring known bots fail the rollout

For experiments, create a conditional override that forces units in this segment to receive the variant you want.

Experiment conditional override mapping bot segment to control variant

Opt out of bot filtering

Statsig configures bot filtering at the project level. Admins can opt out of filtering in the console settings.

Suggest new bots to Statsig

If you discover bots that Statsig doesn't include in the default set, contact Statsig in Slack. You can also ask Statsig to apply your company's internally managed bot list to all bot filtering.

Bot list

Paste this list into the Values field of the Known Bots segment rule.

text

Scomplerbot, WincherBot, fixbot, keys-so-bot, MojeekBot, Gulper Web Bot, Mattermost-Bot, SerendeputyBot, uipbot, WebCrawler, HearsayPDFBot, WRTNBot, BublupBot, InsytfulBot, DingTalkBot, uk_ldfc_renderbot, crawlers, ImagesiftBot, idealo-bot, taboolabot, KlaxoonBot, SemrushBot, archiver/3.1.1 +http://www.archive.org/details/archive.org_bot, StractBot, crawler_eb_germany_2, exabot, DocBase Crawler, co Bot, Superfeedr bot, Pokey_Bot, GooglePlusBot, OtherwebBot, PubMatic Crawler Bot, SiteAuditBot, Gensparkbot, wpbot, archive.org_bot, Audisto Crawler, amazon-product-discovery-bot, Atomseobot, Googlebot-Mobile, hubspot crawler, XoviOnpageCrawler, PerplexityBot, QualifiedBot, YodaoBot, BitSightBot, GG PeekBot, SMTBot, amazonproductbot, FAST-WebCrawler, TwitterCommerceBot, WellKnownBot, PAGEFREEZER CRAWLER,  dbot, htc_botdugls, RavenCrawler, oBot, notebot, ViberBot, KStandBot, scoopit-crawler, SpeechifyBot, Spider_Bot, txt Crawler, net/bot, BugBountyBot, Letianpai_Robot, by fynd.bot, discobot, LineBotWebhook, ahrefsbot, Veoozbot, tkbot, coccocbot, Googlebot-Video, Streamline3Bot, Zoombot, adbeat_bot, msnbot, Nextdoorbot, node DuckAssistBot, redditbot, Xing Bot, DocSearch Crawler, ; bot, Storebot, playwright-bot, 47_safeAreaBottom, online-webceo-bot, SmarshBot, BeeperBot, ChannelBot, BrightEdge Crawler, //boteden, Quantcastbot, SpringserveBot, IAS Crawler, managr-webcrawler, ) Bot, Dragonbot, crawler4j, tyseobotmobile, IVW-Crawler, SEBot, sap-search-web-crawler, AndersPinkBot, Dcard-link-preview-bot, Mediumbot, Light Crawler, dataforseobot, Better Uptime Bot, CCBot, es_bot, DuckAssistBot, SeznamBot, telegrambot, crawler, Jugendschutzprogramm-Crawler, SeoCherryBot, GroupMeBot, HyperMegaBotGettingOnlyHTMLsFromYourWebsite, our-crawler, Slackbot-LinkExpanding, YandexMobileBot, Web-Crawler, PaperLiBot, Swiftbot, Paqlebot, YandexRenderResourcesBot, MetaJobBot, SynologyChatBot, GenomeCrawlerd, robot, StatusCakeBot, node bitlybot, Your robot, Pharosbot, TSMbot, WalluBot, slackbot, AmazonAdBot, AspiegelBot, EzoicBot, TiggeritoBot, eventseekerBot, AwarioBot, Leikibot, Timpibot, like Gecko) bot, Quora-Bot, JobBot, googlebot, PetalBot, eu bot, LinkArchiver twitter bot, DF Bot, Screaming Frog Wise SEO Spider, Clickagy Intelligence Bot, BLP_bbot, bitlybot, WazzupCrawler, web-crawler, pingbot, yoozBot, triptease-bot, Plesk screenshot bot, Magus Bot, node Screaming Frog SEO Spider, YextBot, seobilitybot, tyseobot, applebot, bingbot, GetLocalBot, TwitterBot, rogerbot, Preview Service; bot, traq-ogp-fetcher-curl-bot, seesawbot, Greppr Web Crawler, ResearchBot, web-bot, iAskBot, JobboerseBot, CriteoBot, FandomOpenGraphBot, com feedbot, Amazon-Advertising-ad-standards-bot, MotoMinerBot, peer39_crawler, Discordbot, DuckDuckGo-Favicons-Bot, KeybaseBot, adsbot, Open Graph Bot, emulate-seobots, bountybot, InfobipCrawler, GoogleBot, macox bot, Google-bot, captify-crawler, Robot, aiHitBot, fedistatsCrawler, ExtendedStayBot, NetpeakCheckerBot, com/bots, Automattic Analytics Crawler, Blog Rssbot, Dubbotbot, Rightlander Crawler, ClarityBot, Cookiebot, UOrgTestingBot, AcademicBotRTU, SEMrushBot, Server Crawler, Diffbot, DiscourseBot, chatbot, VirusTotalBot, SaberBot, TZUnfurlBot, Mail.RU_Bot, Monsidobot, YandexAccessibilityBot, preview service; bot, ecoresearchCrawler, PulsePoint-Crawler, DataForSeoBot, petalbot, Xbot, LinkedInBot, GnowitNewsbot, vebidoobot, Bawaab_bot, Brightbot, ClineCrawler, ; Bot, MSIECrawler, MoodleBot, Testcrawler, AASA-Bot, GPTBot, StrapBot, 5) bot, mj12bot, screaming frog seo spider, HubSpot Crawler, COIBotParser, OcelotBot, com crawler, Pinterestbot, VelenPublicWebCrawler, Firefox superpagesbot2, Parser Robot, GrapeshotCrawler, Mediatoolkitbot, am a bot, semrushbot, SearchAtlas Bot, DiffeoBot, IBM-Crawler, spbot, DatoCmsSearchBot, SISTRIX Crawler, bountybotttt, Summalybot, ID bot, node AppleNewsBot, DotBot, TesseractBotAgent, foundeebot, BadooBot, BacklinksExtendedBot, wowLink Crawler, about-crawlers, find-seo-bot, GraphiteBot, Sidetrade indexer bot, BLEXBot, rc-crawler, FacebookBot, nerdybot, Senutobot, Facebot, //botim, WallabyupBot, TurnitinBot,  XBot_Senior, node ZoominfoBot, AhrefsBot, Exabot, PhaverBot, Applebot, TermlyBot, SemjiBot, Space Unfurl Bot, Slackbot, bidswitchbot, //botsin, AdsBot-Google, Morningscore Bot, DuckDuckBot, UptimeRobot, ClaudeBot, naverbookmarkcrawler, PingdomBot, Web Crawler, PlurkBot, node GrowSEOBot, WebExplorerSearchBot, node FullStoryBot, WebwikiBot, bot, policy adbeat_bot, trendictionbot0, ezoicbot, Catrobatbot, AdsTxtCrawlerTP, com/bot, Nigooutbot, PiBot, pinterestbot, http-spiders-bot, Ocarinabot, msnbot-media, AppsFlyerBot, SeobilityBot, Impressumscrawler, SurdotlyBot, cXensebot, Amazonbot, Rankabot, 2ip bot, harsilbot, FullStoryBot, com bot, Rhobot, FreshpingBot, twitterbot, Twitterbot, Caliperbot, Googlebot-Image, osapon ) bot, yandexbot, MJ12bot, Taboolabot, ActiveComplyBot, MixrankBot, 48_safeAreaBottom, compatible; botify, LoomlyBot, Googlebot, ev-crawler, pagefreezer crawler, AwarioSmartBot, iCjobs Stellenangebote, jbot, aixnew_aibot, SemanticScholarBot, Wire LinkPreview Bot, Elastic-Crawler, UCMore Crawler, x28-job-bot, ISSCyberRiskCrawler, AnytypeBot, clever tech bot, LivelapBot, Screaming Frog SEO Spider, RyteBot, SiteGuruCrawler,  XBot, SuperBot, TypetalkBot, RepoLookoutBot, obot, TimeTreeBot, siteauditbot, iASD_SpiderBot, semaltbot, PopeTech-ScanBot, SummalyBot, aka-bot, YandexBot, HatenaBlog-bot, Googlebot-News, yacybot, SiteCheckerBotCrawler, node AwarioSmartBot, turbotime, net-Robot, reurl-bot, Google-Display-Ads-Bot, node CCBot, fr_bot, CapterraBot, seo-audit-check-bot, gptbot, Testomatobot, Snap-URL-Preview (bot, robots, ZumBot, hstspreload-bot, Fedicabot, serpstatbot, Synapse (bot, NetSeer crawler, uk_ldfc_bot, PartnerOptimizer-bot, Scrapbox Bot, domainsbot, NE Crawler, startmebot, VelaBot, SeekportBot, Radius Compliance Bot, AdkernelTopicCrawler, Linkbot, AppleNewsBot, Jones Searchbot, DropboxPreviewBot, ZoominfoBot, edansbot, Baiduspider-render, Python Requests, YisouSpider, OAI-SearchBot, AliyunSecBot, Baiduspider, Bytespider, TikTokSpider, Claude-SearchBot, Qwantbot, Thinkbot, ABEvalBot, HanaleiBot, OpenindexSpider, VamosBot, StartmeBot, Bingbot, parisbot, BetaLiveCrawlBot, MetaBot, CharSiuBot, PlagAwareBot, io/bot, ShowUpCrawler, ScraperBot, Monibot, ShellBot, IbouBot, MatchboxBot, CookieYesbot, Stripebot, Clearscopebot, quillbot, Sogou web spider, Checker Spider,

Was this helpful?