Should you block GPTBot? OpenAI's AI training crawler
GPTBot is operated by OpenAI. It collects public web content to train OpenAI's future models. Blocking it keeps your words out of the next training set; it does not remove you from today's AI answers.
Who should block GPTBot, and who should not?
Block it if you do not want your writing used to train OpenAI's models. The block costs nothing in ChatGPT search, which runs on a separate crawler, OAI-SearchBot, under a separate setting. That makes GPTBot one of the cheapest blocks in this directory. The case for leaving it open is indirect: you may want future models to know your brand, product names, or definitions without a search step. OpenAI documents no traffic benefit to allowing it.
The verified facts
| User-agent token | GPTBot/1.4 |
|---|---|
| Full user agent in logs | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
| Operator | OpenAI |
| Purpose | Model training crawlers |
| Honors robots.txt | Yes, per the operator's documentation. |
| Published IP ranges | openai.com/gptbot.json |
What does OpenAI’s documentation add?
- OpenAI says each robots.txt setting is independent: a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep content out of training.
- If a site allows both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both purposes to avoid crawling twice.
- When it fetches robots.txt, GPTBot may add a robots.txt marker to its user-agent string, so those requests are easy to spot in logs that omit the path.
How do you block GPTBot?
Add this to your robots.txt:
User-agent: GPTBot
Disallow: /Once you give a crawler its own group, it stops reading your User-agent: * rules, so run the whole file through the AI crawler robots.txt tester before you deploy it.
How do you tell a real GPTBot request from a fake one?
A user-agent match proves nothing, since any client can send GPTBot/1.4. Check the IP against the published range file above, re-fetched on a schedule. A request that fails is not OpenAI. The log verification guide has a script for the IP check and a worked reverse-DNS example, or paste the IP into the AI crawler IP verifier, which runs these checks against OpenAI’s current data.
What does blocking GPTBot cost you?
OpenAI treats the block as a signal that your content should not be used to train its generative AI foundation models. ChatGPT search and user-requested visits are controlled by other tokens.
What else does OpenAI document?
OpenAI also documents OAI-SearchBot, ChatGPT-User and OAI-AdsBot, each with its own robots.txt token. How OpenAI’s crawlers fit together.
Which crawlers in the same group should you decide on at the same time?
A robots.txt group for GPTBot does nothing to crawlers with other tokens. In the same group, the directory also covers ClaudeBot (Anthropic), Applebot-Extended (Apple), AI2Bot (Allen Institute for AI) and MistralAI-Training (Mistral), each with its own token and its own documented cost of blocking.