Should you block Amzn-SearchBot? Amazon's search crawler
Amzn-SearchBot is operated by Amazon. It crawls and indexes web content for search experiences in Amazon products and services, including Alexa.
Who should block Amzn-SearchBot, and who should not?
Allow it if you want to be eligible for Amazon search experiences such as Alexa answers; Amazon says it does not collect training data, so the usual training objection does not apply. Note the fallback: with no Amzn-SearchBot group, it follows whatever you set for other search bots, so a site that blocks everything except Googlebot and Bingbot may still admit it. Name it explicitly if you want it out.
The verified facts
| User-agent token | Amzn-SearchBot/0.1 |
|---|---|
| Full user agent in logs | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36 |
| Operator | Amazon |
| Purpose | Search and AI-search indexers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Published IP ranges | developer.amazon.com/amazonbot/searchbot-ip-addresses/ |
What does Amazon’s documentation add?
- If robots.txt does not mention Amzn-SearchBot but allows other search bots, Amazon says it crawls under the rules given to those bots.
How do you block Amzn-SearchBot?
Add this to your robots.txt:
User-agent: Amzn-SearchBot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real Amzn-SearchBot request from a fake one?
A user-agent match proves nothing, since any client can send Amzn-SearchBot/0.1. Check the IP against the published range file above, re-fetched on a schedule. A request that fails is not Amazon. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against Amazon’s current data.
What does blocking Amzn-SearchBot cost you?
Content is not eligible to appear in Amazon search experiences such as Alexa.
What else does Amazon document?
Amazon also documents Amazonbot and Amzn-User, each with its own robots.txt token. How Amazon’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for Amzn-SearchBot does nothing to crawlers with other tokens. In the same group, the directory also covers YouBot (You.com), Diffbot (Diffbot), PetalBot (Huawei), MistralAI-Index (Mistral), LinerBot (Liner) and OAI-SearchBot (OpenAI), each with its own token and its own documented cost of blocking.