Free robots.txt Checker for AI Crawlers: Read Your File Bot by Bot

A robots.txt check answers one narrow question: does your file give a named AI crawler permission to fetch a URL? That question matters because the rules are per bot. Allowing one crawler says nothing about another.

The check tells you three things:

• Which AI bots your file names, and which fall back to the User-agent: * group.
• Whether each bot is allowed or disallowed for a specific path, such as your homepage or pricing page.
• Whether a blanket AI block has removed a search bot you meant to keep.

That last error is the common one. A site blocks "all AI" to stop training, and quietly loses ChatGPT search eligibility in the process. For the full list of bot names and what each controls, see our per-bot guide to robots.txt for AI crawlers.

Each AI company runs more than one crawler. Read every line of your file against this split (as of October 2026, per each operator's documentation).

OpenAI states that "each setting is independent of the others", so you can block GPTBot and still allow OAI-SearchBot. If that trade-off is your question, read whether to block GPTBot and keep ChatGPT search.

Google is different. There is no robots.txt token for AI Overviews or AI Mode. Googlebot is the control there, and Google-Extended governs Gemini Apps training and grounding without affecting Search.

You need a terminal and five minutes.

1. Fetch the file as a browser would: curl -s https://example.com/robots.txt. Confirm it returns your rules, not an HTML error page.
2. Fetch it again as an AI crawler: curl -s -o /dev/null -w "%{http_code}" -A "OAI-SearchBot" https://example.com/robots.txt. A 200 means the bot can read the file. A 403 means it cannot even see your rules. Our robots.txt reachability check covers this step in more depth.
3. List every User-agent: line. Note which AI bots have their own group and which fall through to *.
4. For each bot in the table above, find the group it will follow (see the next section).
5. Inside that group, test the paths that matter: /, /pricing, /blog/, your product pages.
6. Write down the result per bot: allowed, disallowed, or no rule (which means allowed).

Here is a file that blocks training and allows search for OpenAI:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /admin/

For a longer walkthrough with more bots, use the playbook for testing robots.txt per AI bot.

The Robots Exclusion Protocol (RFC 9309) sets the rules most crawlers follow. Three of them decide almost every result.

A bot follows its own group only

If a group names the bot, the bot uses that group and ignores User-agent: *. In the example above, OAI-SearchBot never reads the Disallow: /admin/ line. If no group names the bot, it falls back to *. Product names are matched case-insensitively, a detail covered in whether AI bot names are case-sensitive.

The longest matching path wins

Within a group, the rule whose path matches the most characters applies. Disallow: /blog/ plus Allow: /blog/launch means /blog/launch is allowed and the rest of /blog/ is not.

Ties go to Allow

When an Allow and a Disallow match the same length, the protocol says the Allow should be used.

No matching rule means the path is allowed. An empty Disallow: line also allows everything.

A clean robots.txt is permission, not access. Three gaps remain.

• Firewalls and CDNs. OpenAI's help centre says eligibility for ChatGPT search needs both an OAI-SearchBot allow and a host or CDN that lets OpenAI's published searchbot IP addresses through. A bot-management rule can return 403 while robots.txt says yes. Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains and free-tier customers from 15 September 2026, so a site can change without anyone editing it. Run the firewall AI bot test next.
• User-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules."
• What happens after access. Allowing a bot buys eligibility. It does not guarantee that the bot fetches your page, that the engine retrieves it for a query, or that the answer cites you.

The one first-party statement tying access to citation is Meta's: allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses."

Reading a long file by hand for a dozen bots is slow and easy to get wrong. The robots.txt Checker for AI Crawlers fetches your file and resolves Allow and Disallow for each named AI bot, with training and search bots kept apart.

Treat its result as step one. Then confirm the bots can actually reach your pages with the free AI crawler checker, which requests a page as each crawler and shows real status codes.