Should you block YouBot? You.com's search crawler
YouBot is operated by You.com. It powers the You.com search engine, discovering and indexing pages for its search results.
Who should block YouBot, and who should not?
YouBot is the easiest crawler in this directory to verify and to throttle: a single published /24, reverse DNS, signed requests, Crawl-delay support, and a 30-minute robots.txt cache. That makes a block or a slowdown low-effort and reliable. Whether to allow it depends on whether You.com's search users are an audience for you; the crawl feeds its search results. Most sites lose little either way, so throttle before you block.
The verified facts
| User-agent token | YouBot/1.0 |
|---|---|
| Full user agent in logs | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36 |
| Operator | You.com |
| Purpose | Search and AI-search indexers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Reverse DNS ends in | search.you.com |
What does You.com’s documentation add?
- You.com recommends cryptographic verification: YouBot signs requests under Cloudflare's Web Bot Auth standard, with public keys at you.com/.well-known/http-message-signatures-directory.
- Genuine requests come from 68.67.112.0/24, and reverse DNS names take the form youbot-68-67-112-106.search.you.com.
- It honors Crawl-delay, slows down on HTTP 429, and caches robots.txt for 30 minutes, so changes apply quickly.
How do you block YouBot?
Add this to your robots.txt:
User-agent: YouBot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real YouBot request from a fake one?
A user-agent match proves nothing, since any client can send YouBot/1.0. Reverse-resolve the IP, check the name ends in search.you.com, and forward-resolve that name back to the same IP. A request that fails is not You.com. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against You.com’s current data.
What does blocking YouBot cost you?
Your pages drop out of You.com search results.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for YouBot does nothing to crawlers with other tokens. In the same group, the directory also covers Diffbot (Diffbot), PetalBot (Huawei), MistralAI-Index (Mistral), LinerBot (Liner), OAI-SearchBot (OpenAI) and Claude-SearchBot (Anthropic), each with its own token and its own documented cost of blocking.