Bytespider: what this crawler is

99Visibility Research · Updated October 21, 2026 · 4 min read

Short answer

Bytespider is a web crawler user agent, commonly attributed to ByteDance, that is often grouped with AI bots in robots.txt and firewall lists. Independent re-testing in June 2026 found it does not execute JavaScript, so it only sees content in your raw HTML. Treat it as one bot with its own rule. Allowing or blocking it has no documented effect on ChatGPT, Perplexity, Claude or Google.

Key takeaways

Bytespider is a crawler user agent that sites often lump into a generic AI bot block.
As of a June 2026 re-test, Bytespider does not execute JavaScript, so client-side content is invisible to it.
robots.txt rules are per bot: a Bytespider rule does not change how OAI-SearchBot, PerplexityBot or Claude-SearchBot treat your site.
No first-party evidence links allowing Bytespider to citations in any AI answer engine.
A user agent string can be faked, so verify real hits before acting on log data.

What Bytespider is

Bytespider is the name a crawler sends in its user agent string when it requests your pages. It is commonly attributed to ByteDance, the company behind TikTok. Security vendors and robots.txt templates often list it alongside AI crawlers such as GPTBot and ClaudeBot.

Our evidence base holds no first-party document from its operator. That means we can't tell you what the fetched content is used for, how often it returns, or which products it feeds. What we can say is how it behaves when it fetches a page, and how robots.txt applies to it.

How it works

Bytespider requests a URL and reads the HTML the server sends back. It does not run the page's JavaScript. An independent re-test in June 2026 confirmed this for a set of crawlers that includes GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider. Vercel's crawler study reached the same conclusion for the ChatGPT and Claude crawlers: they download JavaScript files but "don't execute them."

What this means in practice:

Text, prices or product names injected after hydration are missing from what Bytespider stores.
Content in the initial HTML response is visible to it.
Tabs, accordions and "load more" sections that fetch content on click are invisible too.

If you want to check this on your own pages, compare the raw HTML with the rendered page. Our page on seeing your site the way an AI crawler sees it walks through how.

Control works through robots.txt, matched on the user agent token:

User-agent: Bytespider
Disallow: /

Token matching has its own quirks, covered in whether AI bot names are case-sensitive.

Why it matters for AI visibility

Bytespider matters mostly because of how sites handle it, not because of any documented visibility payoff. The common real-world error is a blanket AI block. A site copies a list of "AI bots" into robots.txt or a firewall rule, and that list includes both Bytespider and retrieval bots like OAI-SearchBot. OpenAI documents that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."

So the decision should be made one bot at a time:

List every AI user agent your robots.txt and firewall mention.
For each, note what its operator documents it does. OpenAI's split between GPTBot, OAI-SearchBot and ChatGPT-User is a good model, explained in OpenAI crawlers, robots.txt and firewalls.
Decide on Bytespider on its own terms, such as server load or your content policy.
Confirm the bots you want can actually reach you. A firewall can return 403 to a bot that robots.txt allows. The AI Crawler Checker requests a page as each AI crawler and shows which ones are turned away.

If Bytespider is not named in your file at all, the general rules apply. See whether an unlisted AI bot is allowed.

Common misconceptions

"Blocking Bytespider hides me from AI search." No engine documents that. ChatGPT search, Perplexity and Claude search each use their own bots, and each robots.txt rule is independent.
"Allowing Bytespider gets me cited." No first-party source ties Bytespider access to citations anywhere. Allow it or block it for your own reasons, not for visibility.
"robots.txt settles it." robots.txt is a request, not an enforcement layer. Cloudflare notes that site owners are unsure whether newer or less transparent crawlers respect it. If compliance matters to you, enforce at the firewall.
"A log line saying Bytespider is Bytespider." Anyone can send that string. Before you build rules on log data, verify real AI crawlers against spoofed user agents.

Frequently asked questions

What does the Bytespider user agent string look like?

It contains the token "Bytespider", which is what robots.txt matches against in a User-agent line. The full string varies and can change, so match on the token rather than the whole string. Because any client can send this token, a log entry alone doesn't prove the request came from the real crawler. Verify it through the operator's published methods or reverse DNS before writing rules around it.

Should I block Bytespider?

That depends on your goals, not on AI visibility. No source in our evidence base links Bytespider to citations in ChatGPT, Perplexity, Claude or Google AI features. Common reasons to block it are server load and content-use policy. Whatever you choose, write a rule for Bytespider alone. Don't fold it into a blanket AI block that also catches retrieval bots like OAI-SearchBot.

Does Bytespider render JavaScript?

No, according to an independent re-test in June 2026, which found it does not execute JavaScript. It reads only the HTML your server returns. Any content that appears only after scripts run is invisible to it. The same is true of most AI crawlers tested, including OAI-SearchBot, ClaudeBot and PerplexityBot. Googlebot and AppleBot are the exceptions that render.

Does blocking Bytespider in robots.txt affect Googlebot or other bots?

No. robots.txt rules apply to the user agent group they are written under. A User-agent: Bytespider block with Disallow: / affects only crawlers that match that token. Googlebot, OAI-SearchBot and every other bot follow their own groups, or the wildcard group if they have none. Check that your file doesn't also contain a broad wildcard rule you didn't intend.

Will Bytespider obey robots.txt?

Its operator has not published documentation in our evidence base, so we can't confirm its compliance. Cloudflare notes that site owners are often unsure whether newer or less transparent crawlers respect robots.txt. If you need a block to hold, enforce it at your CDN or firewall as well, then check your logs to see whether verified requests stop.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.