GPTBot, OAI-SearchBot and ChatGPT-User: what each OpenAI bot does, and what to put in robots.txt

On 10 September 2026 we reviewed the robots.txt on novatechray.com, and every block in it carries a comment.

On 9 October 2026 we reread OpenAI's crawler page against it, line by line. Two of our own comments said more than OpenAI does.

That is the reason for this post. Google's autocomplete, checked the same day, finishes "gptbot" with "gptbot user agent", "gptbot robots.txt" and "gptbot crawler", and finishes "oai-searchbot" with "oai searchbot vs chatgpt user". People are trying to work out which bot is which before they touch the file.

Here is the answer from the source, with the exact strings, and three files you can paste.

Step 1: three bots, three jobs

OpenAI's Overview of OpenAI Crawlers, read on 9 October 2026, lists four user agents. Three of them visit ordinary websites.

GPTBot crawls content that may be used to train OpenAI's foundation models. Disallowing it tells OpenAI not to use your content for that training.

OAI-SearchBot is for search. It surfaces websites in ChatGPT's search features, and OpenAI recommends allowing it if you want to appear there.

ChatGPT-User is a visit made because a person asked ChatGPT, or a custom GPT, something that needed your page. OpenAI says it does not crawl the web automatically.

The fourth, OAI-AdsBot, only visits landing pages submitted as ChatGPT ads, and OpenAI says its data is not used for training. If you do not buy ChatGPT ads, it has no reason to visit you.

The line that matters most on the page is short: "Each setting is independent of the others."

Step 2: the complication in our own file

Our robots.txt comment above the answering bots said blocking any one of them removes you from that surface entirely.

OpenAI's page says something narrower. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, but can still appear as navigational links.

Our comment above the user-initiated fetchers said ChatGPT-User ignores robots.txt. OpenAI's wording is softer: because a user starts these actions, robots.txt rules may not apply.

Neither error changed what the file does. Every rule in it is an Allow. But a comment is advice to whoever edits the file next, and ours overstated both points.

So we corrected both comments on 9 October 2026, and this post is written from OpenAI's wording, not ours.

Step 3: the exact user-agent strings

These are copied from OpenAI's page as read on 9 October 2026. For GPTBot and OAI-SearchBot it calls them example strings and says the version number may change.

GPTBot:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

OAI-SearchBot:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

ChatGPT-User, listed as the full string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Two details people miss. OAI-SearchBot's string starts like a desktop Chrome browser, so a log filter that drops anything containing "Chrome" drops it too.

And when GPTBot or OAI-SearchBot fetches robots.txt itself, OpenAI says it may add a "robots.txt" marker to the string. That helps when your logs record agents but not paths.

Autocomplete still offers "gptbot 1.3" and "oai searchbot 1.3". Version numbers move. Match on the token, never the number.

Step 4: GPTBot is the wrong lever

This is the decision most owners get backwards.

If your worry is training, GPTBot is your switch. If your worry is whether ChatGPT names you when a customer asks, GPTBot is the wrong lever, because OpenAI points search opt-outs to OAI-SearchBot, not GPTBot.

OpenAI gives the exact pairing as its own example: allow OAI-SearchBot to appear in search while disallowing GPTBot to keep content out of training. It also says that if you allow both, it may use the results of one crawl for both purposes to avoid crawling twice.

So blocking GPTBot buys you a training opt-out and costs you nothing in ChatGPT search, according to OpenAI. Blocking OAI-SearchBot costs you the search answer, which is the surface our post on getting ChatGPT to recommend your business is about.

And ChatGPT-User sits outside both decisions. OpenAI says it is not used to decide what appears in search and points you to OAI-SearchBot for search opt-outs.

Step 5: three robots.txt files for the three common choices

One rule first: a crawler obeys one group.

Google's robots.txt documentation, read on 9 October 2026, puts it plainly: "User agent specific groups and global groups (*) are not combined." Name a bot and it stops reading your wildcard rules, so repeat every Disallow you still want inside its group.

Our AI crawlers and robots.txt guide covers the other providers. These three are OpenAI only.

Choice A: allow everything. Right for a clinic, a restaurant, a hotel or any site whose job is to be found.

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

No OpenAI line is needed. All three bots fall under the wildcard.

Choice B: stay in ChatGPT search, opt out of training.

User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

OAI-SearchBot and ChatGPT-User still read the wildcard group, including the /admin/ rule.

Choice C: out of training and out of ChatGPT search answers.

User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

You can still show up as a navigational link, per OpenAI. And OpenAI says a search change takes about 24 hours to register after you update the file.

None of the three names ChatGPT-User. Adding a Disallow for it is allowed, and OpenAI's own wording says it may not be honoured.

Step 6: the published IP lists, checked

robots.txt is a request. To verify a visitor really is OpenAI, check the IP against OpenAI's lists.

We fetched all four on 9 October 2026. Each one loaded with HTTP 200 and parsed as JSON with a creationTime field and a list of IPv4 prefixes.

• openai.com/gptbot.json: 18 prefixes, creationTime 22 September 2026.
• openai.com/searchbot.json: 39 prefixes, creationTime 2 January 2026.
• openai.com/chatgpt-user.json: 234 prefixes, creationTime 7 October 2026.
• openai.com/adsbot.json: 2 prefixes, creationTime 12 May 2026.

Two things showed up in that fetch. The ChatGPT-User list was regenerated two days before we read it, so a firewall allowlist copied once will drift. Pull it on a schedule.

And that same file came back with a content type of application/octet-stream, while the other three were served as application/json. A script that checks the content type before parsing will reject it. Parse it anyway.

If you block by IP at the edge instead, read our post on whether Cloudflare is blocking ChatGPT from your website first. An edge rule refuses the request before robots.txt is ever read.

Step 7: check what is live

We sent a request to our own guide on novatechray.com with each of the three strings above on 9 October 2026. All three got HTTP 200.

That is the whole test, and it is worth running on yours. Fetch your live /robots.txt, not the copy in your repository, then request one page with each string and read the status code.

If your site hosts audio or video, remember what the bot gets when it arrives: text. VALORAE Arc's post on why AI assistants read the transcript, not the podcast covers that half.

NovaTechRay will update this post when OpenAI changes the page. The version numbers alone suggest it will.

The same breakdown for the other two companies is in ClaudeBot, Claude-User and Claude-SearchBot and PerplexityBot and Perplexity-User.

The short version

• Treat GPTBot as the training switch and nothing else.
• Treat OAI-SearchBot as the ChatGPT search switch, and leave it allowed if you want to be named.
• Expect ChatGPT-User to visit when a person asks, since OpenAI says robots.txt may not apply.
• Repeat every Disallow inside any bot group you name.
• Verify bots against OpenAI's four IP lists, and refresh them on a schedule.
• Allow about 24 hours after a robots.txt change before checking ChatGPT search.

This is the work NovaTechRay does at novatechray.com.