Should you block Bingbot? Microsoft's search crawler
Bingbot is operated by Microsoft. It crawls pages for the Bing search index.
Who should block Bingbot, and who should not?
Do not block it on pages you want found. Bing's index reaches beyond Bing: DuckDuckGo, for one, says it sources most of its traditional links from Bing. If Bingbot's load is a problem, use Crawl-delay or the hourly Crawl control in Bing Webmaster Tools rather than a block. Whenever you add a Bingbot section, copy every generic rule into it, because Bing stops reading the * section once it finds its own.
The verified facts
| User-agent token | bingbot |
|---|---|
| Full user agent in logs | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 |
| Operator | Microsoft |
| Purpose | Search and AI-search indexers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Published IP ranges | www.bing.com/toolbox/bingbot.json |
| Reverse DNS ends in | search.msn.com |
What does Microsoft’s documentation add?
- Once Bingbot finds a robots.txt section for itself, Bing says it ignores the generic * section, so repeat the general rules in its section.
- Bing supports Crawl-delay in a Bingbot section, and Bing Webmaster Tools has a Crawl control tool that sets the crawl rate by the hour.
- Bing says search engines cache robots.txt for at least a few hours, so a change can take a few hours to show in its crawling.
- BingPreview, which makes page snapshots, sends the same bingbot user-agent string. Bing says to refresh its IP list daily rather than hardcode ranges.
How do you block Bingbot?
Add this to your robots.txt:
User-agent: bingbot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real Bingbot request from a fake one?
A user-agent match proves nothing, since any client can send bingbot. Check the IP against the published range file above, re-fetched on a schedule. Or reverse-resolve the IP, check the name ends in search.msn.com, and forward-resolve that name back to the same IP. A request that fails is not Microsoft. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against Microsoft’s current data.
What does blocking Bingbot cost you?
Content excluded from the Bing Search index, and from search products that draw on Bing's results, such as the traditional links DuckDuckGo says it largely sources from Bing.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for bingbot does nothing to crawlers with other tokens. In the same group, the directory also covers Applebot (Apple), PerplexityBot (Perplexity), DuckAssistBot (DuckDuckGo), DuckDuckBot (DuckDuckGo), Meta-WebIndexer (Meta) and Amzn-SearchBot (Amazon), each with its own token and its own documented cost of blocking.