Should you block Amazonbot? Amazon's product crawler
Amazonbot is operated by Amazon. It crawls public content to improve Amazon products and services, and Amazon says the content may also train its AI models.
Who should block Amazonbot, and who should not?
Amazon is the one operator here that ties allowing its crawler to a benefit: sites that allow Amazonbot may be eligible for Amazon Content Partners, which lists a +1% affiliate commission boost on eligible sales, hosting credits, and traffic tools (terms at contentpartners.amazon.com). Beyond that, allowing it mostly feeds Amazon's products and possibly its model training. A middle path Amazon documents: allow crawling and add noarchive to pages you do not want used for training. Blocking it does not affect Alexa search, which uses Amzn-SearchBot. Amazon says a settings change may take about 24 hours to apply, and the bot may work from a robots.txt copy up to 30 days old.
The verified facts
| User-agent token | Amazonbot/0.1 |
|---|---|
| Full user agent in logs | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36 |
| Operator | Amazon |
| Purpose | Other product crawlers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Published IP ranges | developer.amazon.com/amazonbot/ip-addresses/ |
What does Amazon’s documentation add?
- Amazon says Amazonbot improves its products and services and may be used to train Amazon AI models.
- Amazon honors the noarchive robots meta tag as do not use the page for model training, so a page can stay crawlable but out of training.
- Amazon says a change may take about 24 hours to apply, but the bot may use a robots.txt copy cached for up to 30 days, and treats an unreachable robots.txt as if none exists. It does not support Crawl-delay.
- Amazon says sites that allow Amazonbot may be eligible for its Content Partners program, which lists a +1% affiliate commission boost on eligible sales, free hosting credits, and AI traffic management tools.
How do you block Amazonbot?
Add this to your robots.txt:
User-agent: Amazonbot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real Amazonbot request from a fake one?
A user-agent match proves nothing, since any client can send Amazonbot/0.1. Check the IP against the published range file above, re-fetched on a schedule. A request that fails is not Amazon. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against Amazon’s current data.
What does blocking Amazonbot cost you?
Content excluded from Amazon product-improvement crawls and AI model training.
What else does Amazon document?
Amazon also documents Amzn-SearchBot and Amzn-User, each with its own robots.txt token. How Amazon’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for Amazonbot does nothing to crawlers with other tokens. In the same group, the directory also covers ImagesiftBot (ImageSift (Hive)), OAI-AdsBot (OpenAI), GoogleOther (Google), FacebookExternalHit (Meta) and Meta-ExternalAds (Meta), each with its own token and its own documented cost of blocking.