Should you block PetalBot? Huawei's search crawler
PetalBot is operated by Huawei. It crawls pages for Petal Search indexing and related Huawei search services.
Who should block PetalBot, and who should not?
Allow it if people using Huawei devices are part of your audience, since PetalBot feeds both Petal search and the recommendations in Huawei Assistant and AI Search. If they are not, a block costs little. Two practical points: an existing index entry can take months to clear after a block, so email petalbot@huawei.com if removal is urgent; and verify requests by checking that the reverse-DNS name ends in aspiegel.com or petalsearch.com and that its forward lookup returns the same IP.
The verified facts
| User-agent token | PetalBot |
|---|---|
| Full user agent in logs | Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot) |
| Operator | Huawei |
| Purpose | Search and AI-search indexers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Reverse DNS ends in | aspiegel.com or petalsearch.com |
What does Huawei’s documentation add?
- Petal says PetalBot builds the index behind Petal search and the content recommendations in Huawei Assistant and AI Search, both powered by Petal Search.
- Huawei's verification steps say the reverse-DNS name should be in aspiegel.com, but the worked example on the same page resolves to petalbot-114-119-128-10.petalsearch.com. Accept a request only when the name ends in aspiegel.com or petalsearch.com and a forward lookup of that name returns the same IP.
- After a block, Petal says it may take several months to clear pages already in its index. Urgent removal requests go to petalbot@huawei.com.
- Petal documents no Crawl-delay support; it says it adjusts crawling to server capacity, site quality, and how often the site updates.
How do you block PetalBot?
Add this to your robots.txt:
User-agent: PetalBot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real PetalBot request from a fake one?
A user-agent match proves nothing, since any client can send PetalBot. Reverse-resolve the IP, check the name ends in aspiegel.com or petalsearch.com, and forward-resolve that name back to the same IP. A request that fails is not Huawei. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against Huawei’s current data.
What does blocking PetalBot cost you?
Petal says your pages become unsearchable in Petal search and in the Huawei Assistant and AI Search services it powers.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for PetalBot does nothing to crawlers with other tokens. In the same group, the directory also covers MistralAI-Index (Mistral), LinerBot (Liner), OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), Googlebot (Google) and Bingbot (Microsoft), each with its own token and its own documented cost of blocking.