Should you block Claude-User? Anthropic's live-fetch agent
Claude-User is operated by Anthropic. It fetches pages on demand when a user's request needs them, so what it reads feeds live answers and citations rather than a training set.
Who should block Claude-User, and who should not?
Leave it open for public pages you want Claude users to be able to read on request. Because Anthropic says Claude-User honors robots.txt, a block here does work, which makes it the right tool for sections you want out of live Claude answers while Claude-SearchBot indexes the rest. Blocking it for training reasons achieves nothing: training is ClaudeBot's job.
The verified facts
| User-agent token | Claude-User |
|---|---|
| Operator | Anthropic |
| Purpose | Live retrieval and user-request fetchers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Published IP ranges | claude.com/crawling/bots.json |
What does Anthropic’s documentation add?
- When someone asks Claude a question, Claude may visit websites as Claude-User. Anthropic says the token lets site owners control which sites these user-initiated requests can reach.
- Anthropic documents no robots.txt exception for Claude-User. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.
How do you block Claude-User?
Add this to your robots.txt:
User-agent: Claude-User
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real Claude-User request from a fake one?
A user-agent match proves nothing, since any client can send Claude-User. Check the IP against the published range file above, re-fetched on a schedule. A request that fails is not Anthropic. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against Anthropic’s current data.
What does blocking Claude-User cost you?
Anthropic says disabling Claude-User stops Claude retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
What else does Anthropic document?
Anthropic also documents ClaudeBot and Claude-SearchBot, each with its own robots.txt token. How Anthropic’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for Claude-User does nothing to crawlers with other tokens. In the same group, the directory also covers Perplexity-User (Perplexity), meta-externalfetcher (Meta), Amzn-User (Amazon), MistralAI-User (Mistral) and ChatGPT-User (OpenAI), each with its own token and its own documented cost of blocking.