Is Cloudflare blocking ChatGPT from your website?

The GEO newsletters have spent this month running the same headline. Cloudflare blocks AI crawlers on 15 September, act now, book a call. Read Cloudflare's own post and the panic mostly evaporates for the businesses we work with. The new default applies to pages that display ads. A dental practice, a twenty-cover restaurant and a boutique hotel do not run ad slots, so there is nothing for the new rule to act on.

The change that does hit those businesses landed fifteen months earlier and nobody sent a newsletter about it.

Is Cloudflare blocking ChatGPT from your website?

Maybe, and the September date is the wrong thing to check.

On 1 July 2026 Cloudflare announced a three-way split of AI traffic into Search, Agent and Training, with new defaults taking effect on 15 September 2026. The wording is specific: "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default." TechCrunch reported the scope as reaching "new Cloudflare customers, new sites set up by existing customers, and all existing free customers." Within that group, the trigger is advertising on the page.

So if you sell dental implants or sea kayaking tours, this one is not your problem.

Your problem is that on 1 July 2025, Cloudflare changed what happens when any new domain is added. The press release describes it as a permission-based model: "Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers," which means "every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out." MIT Technology Review described the same move as Cloudflare defaulting to blocking AI bots on the sites it hosts.

Which means this: if your web developer moved the site behind Cloudflare's free plan at any point in the last fourteen months, and clicked through the onboarding without reading it, AI crawlers may have been blocked from that day. Nobody would have told you. Your Google rankings would not have moved. You would just be absent from ChatGPT, and you would assume it was a content problem.

Search, Agent and Training are three different doors

The new taxonomy is worth understanding even if the September default misses you, because it is the vocabulary the dashboard now uses.

The middle column is Cloudflare's own wording, from its Block AI Bots documentation. The right-hand column is ours, and we are labelling it as ours because Cloudflare has not published a per-bot mapping into the three new categories. Its separate AI crawler reference still uses the older labels and files ChatGPT-User under "AI Assistant".

What ChatGPT-User does is not in doubt. OpenAI's crawler documentation says that "when users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." That is the fetch that happens while a customer sits waiting for an answer about your opening hours.

The trap is obvious once you see the table. An owner reads "allow Search, block the rest" as the cautious middle path, keeps their index presence, and shuts the door on the crawler that runs at the exact moment of the question. Search and Agent are not a hierarchy. They are separate doors.

There is a second-order effect worth knowing before you touch the Training toggle. Help Net Security reports that "if a website blocks Training crawlers, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked, even when Search crawlers are allowed." Blocking Training to keep your copy out of a model can take your Google listing with it.

A Cloudflare block happens before robots.txt is read

This is the mechanical point that makes the whole check different from a robots.txt audit.

robots.txt is a request. A crawler fetches the file, reads it, and chooses to comply. Well-behaved crawlers from the major labs do comply. A Cloudflare edge rule is not a request. It refuses the connection and returns a 403 before your server is involved and before the crawler has any reason to consult a text file. A perfect robots.txt sitting behind a WAF rule that blocks AI bots is a note taped to the inside of a locked door.

There is a quieter version of the same problem. Cloudflare's managed robots.txt feature generates and maintains a robots.txt that, in Cloudflare's words, "instructs known AI crawlers to stay away from your content." If you already have a file, the documentation says Cloudflare "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response."

Read that twice. The file in your repository is not the file at your URL. You can open the robots.txt your agency wrote, see User-agent: * and Allow: /, and be looking at the bottom half of a document whose top half disallows GPTBot.

One more wrinkle in the other direction. OpenAI's documentation notes that for ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply." So for the Agent-class fetch, robots.txt is not the control surface in either direction. The edge rule is.

The two-minute check

Do this from a terminal, not from your browser, and not from the CMS.

First, read the robots.txt your edge serves, which is the only version that exists as far as a crawler is concerned.

curl -s https://yoursite.com/robots.txt

Compare it line for line against the file you think you published. Any Disallow aimed at GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot or a wildcard AI group that you did not write came from the edge.

Second, check that a plain fetch returns your actual page.

curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/

A 403 or a 503 is the answer. So is a 200 that returns a challenge page rather than your content, which you can see by dropping the -o /dev/null.

Be honest about the limit of a curl test. Cloudflare verifies bots by signature and network, not by the user-agent string you type, so pasting OAI-SearchBot's user agent into curl does not reproduce what the real crawler experiences. For that, open AI Crawl Control in the Cloudflare dashboard, which logs which AI services requested your content and lets you set allow or block rules per crawler. It runs on all plans. The log is the evidence. Everything else is inference.

What to change

In your zone's Security settings, Cloudflare's post says the new AI traffic options "are live now, and can be configured by all existing customers in their zone Settings."

For a local business the setting we would choose is Search allowed, Agent allowed, and Training as a genuine business decision rather than a reflex, with the Googlebot side effect in front of you when you decide. Then turn managed robots.txt off if you intend to control that file yourself, because two systems writing one file is how sites end up blocking crawlers nobody meant to block.

If the curl check comes back clean, the cause is somewhere else, and there are only a handful of candidates. Our AI crawlers and robots.txt guide covers the directives and the training-versus-answering split at the file layer, and why a site can rank first on Google and be unknown to ChatGPT walks the other three causes in order.

The version of this article that has been landing in your inbox all month says act before 15 September. The accurate version is that the date does not apply to you, the setting probably does, and finding out costs one terminal command.