AI Crawler robots.txt Tester

How the tester decides

1  Group: the crawler obeys every group that names its token
     (case-insensitive; "GPTBot/1.4" counts as GPTBot).
     No such group: an operator-documented fallback, if any
     (Applebot, ImagesiftBot -> Googlebot). Otherwise the * group.
     Otherwise nothing applies and every path is allowed.
2  Rule: of the Allow and Disallow lines in those groups that
     match the path, the longest pattern wins. A tie goes to Allow.
     "*" matches any characters; a final "$" anchors the end.
3  No matching rule: allowed. /robots.txt: always allowed.
4  Catches: facts from the crawler's own documentation that
     change what the verdict means.

Sources: steps 1 to 3 follow the Robots Exclusion Protocol standard, RFC 9309 as Google applies it in How Google interprets the robots.txt specification (page last updated 2026-08-31, read 2026-09-26). The tester reproduces every rule-order example on that page. Step 4 comes from each crawler’s card, each read from the operator’s own page and dated there. Most AI operators do not publish how their parser handles edge cases such as wildcards. Where one documents a difference, the tester shows it; otherwise it assumes the standard.

Test your robots.txt

Worked example: one file, four surprises

The tester opens with this hypothetical file, tested against /admin/settings:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Disallow: /search

User-agent: bingbot
Allow: /

The owner meant to keep every crawler out of /admin/, keep GPTBot out of everything, and keep Google out of internal search results. The file does something else:

• Googlebot may crawl /admin/. It has its own group, so the * group does not apply to it. Only /search is closed to it.
• Bingbot may crawl /admin/. Same reason: its own group allows everything. Bing’s documentation says so directly.
• Applebot may crawl /admin/ but not /search. There is no Applebot group, and Apple says Applebot then follows the Googlebot rules, not the * rules.
• Google-Extended shows as blocked from /admin/, through the * group. That does not stop a crawl: Googlebot is still allowed. It only tells Google not to use those pages for Gemini.

The fix is to repeat Disallow: /admin/ in the Googlebot and bingbot groups, or to drop those groups if the only thing they add is the /search rule and you are happy to apply it to everyone. Then test /search?q=test and a normal page such as /blog/post to confirm nothing else moved.

What a verdict does not tell you

• Whether your network lets the crawler in. A CDN or firewall bot rule can refuse a crawler that robots.txt allows. OpenAI and Perplexity both ask sites to allow their published IP ranges as well as the token. Anthropic warns that blocking its IPs can stop its bots from reading robots.txt at all.
• When a change takes effect. Google says it generally caches robots.txt for up to 24 hours. OpenAI says about 24 hours for ChatGPT search, DuckDuckGo says 72 hours for DuckAssistBot, and Amazon says its bots may use a copy cached for up to 30 days.
• Whether a page can still appear. Google says a URL blocked by robots.txt can still be indexed if other pages link to it; to keep it out of results, use noindex on a page Google can crawl. To limit what Google’s AI features show from a page, Google points to snippet controls, not robots.txt.
• Which other hosts need the rule. Each host and subdomain serves its own robots.txt, and Anthropic and Semrush both say the rule must be on each subdomain you want covered.

To check that the traffic you see is the crawler it claims to be, use the log verification guide.