Should you block Claude-SearchBot? Anthropic's search bot
Claude-SearchBot is operated by Anthropic. It crawls and indexes web content to improve the relevance and accuracy of Claude search results.
Who should block Claude-SearchBot, and who should not?
Allow it if you want Claude's search to know your pages; the documented cost of a block is lower visibility and accuracy when Claude users search. A site that wants out of training but in search can block ClaudeBot and leave Claude-SearchBot and Claude-User open, the split Anthropic's three separate tokens allow. Block it only for sections you want kept out of Claude's search entirely.
The verified facts
| User-agent token | Claude-SearchBot |
|---|---|
| Operator | Anthropic |
| Purpose | Search and AI-search indexers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Published IP ranges | claude.com/crawling/bots.json |
What does Anthropic’s documentation add?
- Anthropic says it navigates the web to improve search result quality for users, analyzing content to make search responses more relevant and accurate.
How do you block Claude-SearchBot?
Add this to your robots.txt:
User-agent: Claude-SearchBot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real Claude-SearchBot request from a fake one?
A user-agent match proves nothing, since any client can send Claude-SearchBot. Check the IP against the published range file above, re-fetched on a schedule. A request that fails is not Anthropic. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against Anthropic’s current data.
What does blocking Claude-SearchBot cost you?
Anthropic says disabling Claude-SearchBot stops it indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
What else does Anthropic document?
Anthropic also documents ClaudeBot and Claude-User, each with its own robots.txt token. How Anthropic’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for Claude-SearchBot does nothing to crawlers with other tokens. In the same group, the directory also covers Googlebot (Google), Bingbot (Microsoft), Applebot (Apple), PerplexityBot (Perplexity), DuckAssistBot (DuckDuckGo) and DuckDuckBot (DuckDuckGo), each with its own token and its own documented cost of blocking.