Block, allow or price: the AI crawler decision after Cloudflare's September flip

On 15 September 2026 Cloudflare flipped a default, and every recap you read that week was written for publishers.

Here is the scope.

TechCrunch's 1 July 2026 piece put the customer classes at new Cloudflare customers, new sites set up by existing customers, and all existing free customers, but Cloudflare's 15 September post landed somewhere narrower. A new domain is asked whether it earns money from ads. If it does, the recommended settings are Disallow AI Training for Training and Block on pages with ads for Agent. If it does not, everything stays on Allow, and existing domains carry their settings over. On the pricing side, Pay Per Crawl, launched in Cloudflare's own announcement, now sits beside Pay Per Use, which Cloudflare put into beta on 30 September 2026 as a separate model that pays for use rather than access.

That is a negotiation between large content owners and large model labs.

You are running a clinic, a restaurant, a studio or a twelve-page services site with no ad slots anywhere on it. So the flip mostly has nothing to act on, and the decision you still have to make is the one nobody wrote up.

We nearly shipped the wrong version of it, and then we shipped one that was already out of date. The draft of this tree sitting in the NovaTechRay docs in July said block Training, allow Search, because that reads as the cautious middle path. Then we read Cloudflare's own definition of Training, which folds in mixed-purpose crawlers used for both Training and Search, and deleted the branch. On 15 September 2026, two days before this post first ran, Cloudflare shipped that branch as a setting called Disallow AI Training, on all plans, and we missed it.

Cautious was the wrong instinct in July and is a sound one now. There are four moves available to you: allow, disallow training, block or price. Block and price are usually wrong.

Branch 1: does the flip reach you at all

Two conditions, both required.

You have to be adding a new domain, because existing domains carry their settings over. And the site has to earn money from ads, because that is the question Cloudflare's onboarding asks before it recommends anything stricter than Allow.

Most business sites fail the second condition outright.

A menu page, a services page and a contact page carry no ad inventory, so there is nothing for an ad-page default to apply to.

If both conditions miss you, the September date is not your problem and you can stop reading the news coverage. What you still have is a settings panel you have probably never opened. We wrote the diagnostic for that separately in is Cloudflare blocking ChatGPT from your website, and it is worth ten minutes even if the answer turns out to be no.

Branch 2: the answering crawlers, which you allow

This is the branch with a real cost attached, and it is the one people get backwards.

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot are answering crawlers.

They fetch your page so it can be quoted, summarised and cited inside a live answer, which is the entire mechanism by which a stranger asking about your category ends up reading your name.

Block one and you are removed from that surface. Not demoted. Removed.

There is no partial credit, no residual presence, no slow decay you can reverse next quarter.

The engine simply has nothing of yours to retrieve, so it answers using whoever did not block it.

But the reason people block them anyway is that the bot names look like the training bots, and the dashboard groups them under one scary heading. A row that says "AI crawler" invites a single toggle. The toggle is where the damage happens.

So the rule here has no exceptions worth the paragraph it would take to write. Allow all four. If you want the per-directive version at the robots.txt layer rather than the edge layer, the AI crawlers and robots.txt guide has it.

Branch 3: the training crawlers, which are a genuine choice

GPTBot, ClaudeBot and Google-Extended are a different category and a much smaller decision.

Blocking GPTBot does not remove you from ChatGPT. Blocking ClaudeBot does not remove you from Claude.

And Google-Extended is not even a crawler, it is a token controlling whether already-crawled content may be used for Gemini and for grounding, and disallowing it leaves ordinary Google Search untouched.

Which means the training decision is a licensing preference, not a visibility one.

For a hospitality or services site, there is usually nothing on the page worth withholding. Your opening hours are not intellectual property. If you run a genuine content asset, a research library, a course, a set of original guides, blocking training is defensible and costs you almost nothing on the answer surfaces. The one exception is Google-Extended, which Google's crawler documentation says also controls grounding in Gemini Apps, so disallowing it can cost you citations there.

The trap is the mixed-purpose clause, and it got sharper on 15 September 2026. Cloudflare's Training definition covers crawlers used for both Training and Search, and its Block setting now stops Googlebot, Applebot and Bingbot too, search included. So use the option built for this. The launch post calls it Disallow AI Training and the current docs label it Disallow. It publishes a no-training preference in robots.txt and keeps mixed-use crawlers such as Googlebot and Applebot allowed for search. With Bot Preference Sync on, it also blocks other training crawlers at the edge, including the training-only crawlers run by OpenAI and Anthropic. Bing is the gap: Cloudflare says the setting conveys no robots.txt preference to Bing until Microsoft's support launches, and Microsoft's post dates robots.txt training preferences to 1 February 2027.

Branch 4: pricing, which is not available to you

Pay Per Crawl was announced as a marketplace.

The publisher sets a per-request price, and a crawler that has not paid gets refused rather than served, which Cloudflare's Pay Per Crawl post frames as giving content owners a lever they previously did not have. Pay Per Use, which went into beta on 30 September 2026, flips it: the AI company offers a price for a specific use of your content, you accept or decline it in the dashboard, and you get paid when the buyer reports that use.

A marketplace needs a buyer on the other side.

Nobody is bidding for a twelve-page dental site. There is no scarcity, no archive, no unique corpus, so the price you set is not a negotiation, it is a refusal with extra steps. You end up in the same place as Branch 2, with the answering crawlers turned away, except you also believe you are monetising something.

This is the branch where the publisher recap and the business site part company completely.

Pricing is a real strategy if your content is the product. If your content is marketing for a product, pricing it is a way of hiding your own marketing.

Branch 5: what to check after you decide

Whatever you choose, the setting and the outcome are different things and you should confirm both.

Read the robots.txt your edge serves rather than the one in your repository, because a managed file can prepend rules you never wrote.

Update, 9 October 2026: per PPC Land's 8 October report on HasData's re-run, Cloudflare's 15 September change removed its managed no-training lines from the robots.txt of 118 sites, while most of the affected sites it live-tested still blocked GPTBot at the edge. The served file and the edge rule can now disagree in either direction, so check both.

Then check that a plain unauthenticated fetch of your homepage returns your content and not a challenge page.

Then open the crawler log. Cloudflare's AI Crawl Control logs which AI services requested your content, and that log is the only evidence in this whole exercise. Everything else, including this article, is inference.

One honest limit. A curl test with a pasted user-agent string does not reproduce what a verified crawler experiences, because verification runs on signature and network rather than on the string you typed. So a clean curl is a weak pass, not a certificate.

We run this check on every NovaTechRay engagement before we write a word of copy, for a boring reason.

And content work on a site that refuses the crawler is spend with no delivery mechanism attached.

The short version

• Check whether the September default reaches you at all. No ad slots usually means no.
• Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot, always.
• Treat GPTBot, ClaudeBot and Google-Extended as a separate licensing call, and use Cloudflare's Disallow option rather than Block, because Block stops Googlebot and Bingbot too.
• Skip Pay Per Crawl unless your content is the product rather than the advertisement for it.
• Verify against the edge-served robots.txt and a real fetch, not against your repository.
• Read the crawler log afterwards and treat it as the only evidence.

If the branch you land on is unclear, that is the kind of thing NovaTechRay untangles in an hour at novatechray.com.