Bytespider: ByteDance's Crawler, and Why Many Stores Block It
Bytespider is ByteDance's web crawler. Unlike OpenAI's or Google's search bots, blocking it rarely costs a Western store any visibility — and it has a reputation for heavy crawling. Here's the user-agent and the honest call on whether to allow it.
Bytespider is ByteDance's web crawler. Unlike OpenAI's or Google's search bots, blocking it rarely costs a Western store any visibility — and it has a reputation for heavy crawling. Here's the user-agent and the honest call on whether to allow it.
Not every AI crawler is one you want to let in. Bytespider — ByteDance's crawler, the company behind TikTok — is the clearest case where blocking is a defensible, often sensible choice for a Shopify store.
What Bytespider is
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
Bytespider collects web data for ByteDance's products and AI models, which serve primarily the Chinese market (where ChatGPT, Gemini and Perplexity don't operate). It has also been widely reported for very aggressive, high-volume crawling and for historically ignoring robots.txt.
Should a store allow it?
This is the opposite of OAI-SearchBot. For most Western Shopify stores:
- The upside is near zero — ByteDance's AI surfaces aren't where your shoppers are asking for recommendations.
- The downside is real — it can add meaningful server load.
So unlike OpenAI's or Google's search crawlers (which you want open because they decide whether you're recommended), blocking Bytespider is a reasonable performance decision with little visibility cost. If you sell into the Chinese market, weigh it differently.
Just know that, because of the compliance reports, a robots.txt Disallow may need backing up with server- or CDN-level blocking to actually take effect.
The robots.txt
# Many stores block Bytespider — little AI-visibility cost in Western markets
User-agent: Bytespider
Disallow: /
Check what's reaching you
- Run your store through the AI crawler & robot checker — see which bots your robots.txt allows, and which you might want to close.
- Scan your store's AI readiness — make sure that while closing the low-value bots, you've kept the ones that actually recommend you open.
See also: GPTBot · OAI-SearchBot · PerplexityBot · the complete AI crawler & user-agent list.
FoundGPT helps Shopify stores get read, recommended and sold by AI engines. Start with a free AI-readiness scan.