Is Cloudflare blocking ChatGPT from your website?

The GEO newsletters have spent this month running the same headline. Cloudflare blocks AI crawlers on 15 September, act now, book a call. Read Cloudflare's own post and the panic mostly evaporates for the businesses we work with. The new restrictions are recommended only for domains that earn money from ads. A dental practice, a twenty-cover restaurant and a boutique hotel do not run ad slots, so there is nothing for the new rule to act on.

The change that does hit those businesses landed fifteen months earlier and nobody sent a newsletter about it.

Is Cloudflare blocking ChatGPT from your website?

Maybe, and the September date is the wrong thing to check.

On 1 July 2026 Cloudflare announced a three-way split of AI traffic into Search, Agent and Training, with new defaults taking effect on 15 September 2026. The wording is specific: "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default." TechCrunch reported the scope as reaching "new Cloudflare customers, new sites set up by existing customers, and all existing free customers." Within that group, the trigger is advertising on the page. By 15 September the plan had narrowed: Cloudflare's launch post asks a new domain whether it earns money from ads, recommends Disallow AI Training for Training and Block on pages with ads for Agent if it does, and leaves everything on Allow if it does not.

So if you sell dental implants or sea kayaking tours, this one is not your problem.

Your problem is that on 1 July 2025, Cloudflare changed what happens when any new domain is added. The press release describes it as a permission-based model: "Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers," which means "every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out." MIT Technology Review described the same move as Cloudflare defaulting to blocking AI bots on the sites it hosts.

Which means this: if your web developer moved the site behind Cloudflare's free plan at any point in the last fourteen months, and clicked through the onboarding without reading it, AI crawlers may have been blocked from that day. Nobody would have told you. Your Google rankings would not have moved. You would just be absent from ChatGPT, and you would assume it was a content problem.

Search, Agent and Training are three different doors

The new taxonomy is worth understanding even if the September default misses you, because it is the vocabulary the dashboard now uses.

The middle column is Cloudflare's own wording, from its AI Crawler and Agent access policies documentation. The right-hand column is ours, and we are labelling it as ours because Cloudflare has not published a per-bot mapping into the three new categories. Its separate AI crawler reference still uses the older labels and files ChatGPT-User under "AI Assistant".

What ChatGPT-User does is not in doubt. OpenAI's crawler documentation says that "when users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." That is the fetch that happens while a customer sits waiting for an answer about your opening hours.

The trap is obvious once you see the table. An owner reads "allow Search, block the rest" as the cautious middle path, keeps their index presence, and shuts the door on the crawler that runs at the exact moment of the question. Search and Agent are not a hierarchy. They are separate doors.

There is a second-order effect worth knowing before you touch the Training toggle. Help Net Security reports that "if a website blocks Training crawlers, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked, even when Search crawlers are allowed." Blocking Training to keep your copy out of a model can take your Google listing with it. Since 15 September 2026 that no longer has to be the trade. Cloudflare's Disallow AI Training option, on all plans and labelled Disallow in its current docs, publishes a no-training preference in robots.txt and keeps mixed-use crawlers such as Googlebot and Applebot allowed for search. Use it instead of Block. Bing is the gap until Microsoft's robots.txt training preference starts on 1 February 2027.

A Cloudflare block happens before robots.txt is read

This is the mechanical point that makes the whole check different from a robots.txt audit.

robots.txt is a request. A crawler fetches the file, reads it, and chooses to comply. Well-behaved crawlers from the major labs do comply. A Cloudflare edge rule is not a request. It refuses the connection and returns a 403 before your server is involved and before the crawler has any reason to consult a text file. A perfect robots.txt sitting behind a WAF rule that blocks AI bots is a note taped to the inside of a locked door.

There is a quieter version of the same problem. Cloudflare's managed robots.txt feature generates robots.txt directives from your AI policies when Bot Preference Sync is on and at least one policy restricts something. If you already have a file, the documentation says Cloudflare "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response."

Read that twice. The file in your repository is not the file at your URL. You can open the robots.txt your agency wrote, see User-agent: * and Allow: /, and be looking at the bottom half of a document whose top half disallows GPTBot.

Update, 9 October 2026: the managed file is on its way out. PPC Land reported on 8 October that Cloudflare's 15 September update deprecated Managed Robots.txt, and that HasData's re-run of 10,894 sites on 16 September found the managed no-training lines gone from the robots.txt of 118 sites that had carried them.

Sites that wrote their own rules kept them, and 16 of the 21 affected sites in HasData's live test still blocked or challenged GPTBot at the edge. So your file can now say less than your edge enforces, the reverse of the problem above, and the curl check below catches both.

One more wrinkle in the other direction. OpenAI's documentation notes that for ChatGPT-User, "Because these actions are initiated by a user, robots.txt rules may not apply." So for the Agent-class fetch, robots.txt is not the control surface in either direction. The edge rule is.

The two-minute check

Do this from a terminal, not from your browser, and not from the CMS.

First, read the robots.txt your edge serves, which is the only version that exists as far as a crawler is concerned.

curl -s https://yoursite.com/robots.txt

Compare it line for line against the file you think you published. Any Disallow aimed at GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot or a wildcard AI group that you did not write came from the edge.

Second, check that a plain fetch returns your actual page.

curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/

A 403 or a 503 is the answer. So is a 200 that returns a challenge page rather than your content, which you can see by dropping the -o /dev/null.

Be honest about the limit of a curl test. Cloudflare verifies bots by signature and network, not by the user-agent string you type, so pasting OAI-SearchBot's user agent into curl does not reproduce what the real crawler experiences. For that, open AI Crawl Control in the Cloudflare dashboard, which logs which AI services requested your content and lets you set allow or block rules per crawler. It runs on all plans. The log is the evidence. Everything else is inference.

What to change

In your zone's Security settings, open what Cloudflare renamed AI Crawler and Agent access policies on 4 November 2026. Set the overall Agents policy there. Since that date you can also give each supported personal agent its own setting: Allow, Block on pages with ads, or Block.

For a local business the setting we would choose is Search allowed, Agent allowed, and for Training either Allow or Disallow, which states a no-training preference without the Googlebot side effect that makes Block a bad trade. If you choose Disallow, leave Bot Preference Sync on, because that is what publishes the preference. Otherwise turn it off if you intend to control that file yourself, because two systems writing one file is how sites end up blocking crawlers nobody meant to block.

If the curl check comes back clean, the cause is somewhere else, and there are only a handful of candidates. Our AI crawlers and robots.txt guide covers the directives and the training-versus-answering split at the file layer, and why a site can rank first on Google and be unknown to ChatGPT walks the other three causes in order.

The version of this article that has been landing in your inbox all month says act before 15 September. The accurate version is that the date does not apply to you, the setting probably does, and finding out costs one terminal command.