Should you block SemrushBot? Semrush's SEO crawler
SemrushBot is operated by Semrush. It crawls for Semrush's SEO and marketing database; readers never see its fetches.
Who should block SemrushBot, and who should not?
Blocking it costs readers nothing, but it does not do what many site owners expect. Your backlink profile stays visible in Semrush, because links to you are found on the linking sites; what disappears is the content and outbound links of your own pages. Most sites should throttle instead: a Crawl-delay of up to 10 seconds cuts the load and keeps your pages current in the tool. Block it outright when crawl load is a real cost, and add the tool-specific tokens if you want every Semrush crawler out.
The verified facts
| User-agent token | SemrushBot |
|---|---|
| Operator | Semrush |
| Purpose | SEO and marketing tool crawlers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Verification | No IP list published; Semrush asks sites not to block it by IP because it uses no consecutive IP blocks |
What does Semrush’s documentation add?
- The SemrushBot token covers the backlink index. Other Semrush tools crawl under their own tokens, each blocked separately: SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, SemrushBot-OCOB, SemrushBot-FT, RyteBot, and SemrushBot-ESI.
- It honors Crawl-delay up to 10 seconds; higher values are cut to 10. With no crawl-delay set, it adjusts its rate to your server load.
- Your robots.txt must return HTTP 200. On a 4xx response SemrushBot assumes there are no restrictions; on a 5xx it stops crawling the whole site.
- Semrush says it can take up to one hour or 100 requests to notice a robots.txt change, and each subdomain needs its own robots.txt file.
How do you block SemrushBot?
Add this to your robots.txt:
User-agent: SemrushBot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
What does blocking SemrushBot cost you?
Semrush stops reading your pages, so their content and outbound links drop out of its backlink index. Links that other sites point at you are still found on those sites.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for SemrushBot does nothing to crawlers with other tokens. In the same group, the directory also covers AhrefsBot (Ahrefs), MJ12bot (Majestic) and DataForSeoBot (DataForSEO), each with its own token and its own documented cost of blocking.