Fixing Shopify Crawl Issues: Using Web Bot Auth with Screaming Frog and Sitebulb
E-commerce SEO on the Shopify platform has long presented a unique technical challenge: mass crawling. Due to platform-level rate limits, SEO professionals frequently ran into "Too Many Requests" errors when trying to perform deep, large-scale audits with popular tools like Screaming Frog and Sitebulb.
The good news? Shopify has rolled out a new feature called Web Bot Auth, a game-changer for e-commerce SEO practitioners.
The Shopify Crawl Problem Solved
The root of the issue was that standard SEO crawlers often behaved too aggressively for Shopify's default settings, leading to blocks and incomplete audits. The new Web Bot Auth feature directly addresses this by creating an official authorization pathway.
What Web Bot Auth Does:
The feature allows site owners to formally authorize known, legitimate SEO crawlers—specifically mentioning tools like Screaming Frog, Sitebulb, and DeepCrawl—to crawl the site effectively without triggering the platform’s protective rate limits.
Steps to Configure Web Bot Auth
Chris Lever's valuable resource details the necessary steps for seamless integration, making this a must-read for anyone performing e-commerce SEO on the Shopify platform.
Here is a simplified overview of the process:
-
Enable the Feature: Locate and activate the Web Bot Auth settings within your Shopify administrative panel.
-
Authorize Your Crawler: Use the interface to select and authorize your preferred SEO crawler (e.g., Screaming Frog or Sitebulb). This typically involves generating a unique token or key.
-
Configure the SEO Tool: Paste the unique authorization token/key into the appropriate authentication settings of your SEO crawler.
-
Start the Crawl: Rerun your audit. The crawler will now communicate its authorized status to the Shopify server, allowing for a deep, stable crawl at scale.
This resource is an essential component of an effective Local SEO and e-commerce strategy, ensuring that technical audits are accurate and complete. If you are struggling with crawl budget or "Too Many Requests" errors on a Shopify site, implementing Web Bot Auth is the immediate solution for reliable, large-scale data collection.
