Should you block meta-externalfetcher?
meta-externalfetcher is operated by Meta. It fetches individual links at a user's request, including when Meta's AI navigates a site to complete a task for someone.
Who should block meta-externalfetcher, and who should not?
A robots.txt block here is mostly a statement of preference, since Meta says this fetcher may ignore it. If the concern is load or access to private pages, fix it where it can be enforced: put private content behind a login and rate-limit at the CDN. Meta publishes no IP list, so a firewall rule usually has to match the user-agent string, which any client can send or omit. For public pages you want people to reach through Meta's assistant, leave it open.
The verified facts
| User-agent token | meta-externalfetcher/1.1 |
|---|---|
| Full user agent in logs | meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| Operator | Meta |
| Purpose | Live retrieval and user-request fetchers |
| Honors robots.txt | Partly. The operator says that some requests may bypass robots.txt. |
| Verification | No official IP ranges published |
What does Meta’s documentation add?
- Meta says this crawler may bypass robots.txt because it performs fetches a user requested. A robots.txt rule records your preference; it does not enforce it.
- Meta also lists evaluating and improving agentic AI capabilities as a use, so these fetches are not only one-off page reads for a person.
How do you block meta-externalfetcher?
Add this to your robots.txt:
User-agent: meta-externalfetcher
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
Because this bot may bypass some robots.txt directives, the rule does not enforce every request. Enforcement requires blocking at the CDN or firewall, though without a published IP list that means user-agent matching only.
What does blocking meta-externalfetcher cost you?
Meta users may be unable to have Meta's AI open your pages for them, to the extent the fetcher honors the block. Meta says it may bypass robots.txt because the fetch was requested by a user.
What else does Meta document?
Meta also documents meta-externalagent, FacebookExternalHit, Meta-WebIndexer and Meta-ExternalAds, each with its own robots.txt token. How Meta’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for meta-externalfetcher does nothing to crawlers with other tokens. In the same group, the directory also covers Amzn-User (Amazon), MistralAI-User (Mistral), ChatGPT-User (OpenAI), Claude-User (Anthropic) and Perplexity-User (Perplexity), each with its own token and its own documented cost of blocking.