Should you block FacebookExternalHit? Meta's link previewer
FacebookExternalHit is operated by Meta. It fetches and caches the title, description, and thumbnail for links shared on Facebook, Instagram, and Messenger.
Who should block FacebookExternalHit, and who should not?
Leave it open on every page you want people to share. Blocking it does not stop sharing; it breaks the preview, so links to your site appear on Facebook, Instagram, and Messenger without a proper title, description, or image. The common reason to block it is a private or staging area, and there robots.txt is the wrong tool, because Meta says this crawler may bypass robots.txt for security checks. Put those pages behind authentication instead.
The verified facts
| User-agent token | facebookexternalhit/1.1 |
|---|---|
| Full user agent in logs | facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) |
| Operator | Meta |
| Purpose | Other product crawlers |
| Honors robots.txt | Partly. The operator says that some requests may bypass robots.txt. |
| Verification | No official IP ranges published |
What does Meta’s documentation add?
- It must be able to fetch the page within a few seconds, or Facebook cannot display the preview.
- Open Graph tags must appear within the first 1 MB of the page, and the server must support gzip and deflate encoding.
- Meta says it might bypass robots.txt when it runs security or integrity checks, such as scanning for malware.
- After you fix a page, Meta's Sharing Debugger tool or the Sharing API forces a fresh crawl, so an old preview does not linger.
How do you block FacebookExternalHit?
Add this to your robots.txt:
User-agent: facebookexternalhit
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
Because this bot may bypass some robots.txt directives, the rule does not enforce every request. Enforcement requires blocking at the CDN or firewall, though without a published IP list that means user-agent matching only.
What does blocking FacebookExternalHit cost you?
Shared links may not show an up-to-date title, description, or thumbnail in Meta apps.
What else does Meta document?
Meta also documents meta-externalagent, meta-externalfetcher, Meta-WebIndexer and Meta-ExternalAds, each with its own robots.txt token. How Meta’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for facebookexternalhit does nothing to crawlers with other tokens. In the same group, the directory also covers Amazonbot (Amazon), ImagesiftBot (ImageSift (Hive)), OAI-AdsBot (OpenAI) and GoogleOther (Google), each with its own token and its own documented cost of blocking.