Topic

The technical work that decides whether an AI engine can read you

Short answer

Two conditions decide most of it. The answering crawlers have to get through your CDN and your robots.txt, which means allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot, and the content they came for has to exist in the raw HTML rather than being assembled by JavaScript in the browser. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are a separate and lower-stakes decision, and blocking the wrong one removes you from answers while protecting nothing. Schema is worth having because it states filterable attributes as data, though Ahrefs' matched study of 1,885 pages that added JSON-LD against 4,000 controls found no lift in AI citations, and llms.txt is close to inert for discovery, with Ahrefs finding 97 percent of the published files it located across 137,210 domains had received zero requests.

Crawler access, server-rendered HTML, schema and llms.txt: which technical work an answer engine needs, and which of it does nothing at all.

The technical half of this work is unglamorous, cheap and finite, and it is also the half that most often explains why a business is missing from AI answers entirely. A menu or a price list that only exists after JavaScript runs is invisible to the crawlers behind ChatGPT, Claude and Perplexity, and that single template decision outweighs every content tactic sold on top of it. NovaTechRay puts the crawler and rendering checks first because they are binary: either the page can be read or nothing else you do matters. The part we will concede, which most vendors will not, is that two of the most-sold technical deliverables in this field, schema as a citation lever and llms.txt, have evidence pointing at roughly nothing. Ship them if you want them for other reasons, and do not let anyone bill you for them as the reason you get cited.

In this topic

Technical

The shared robots.txt AI training line means one thing at Microsoft and, as Google plans it, two at Google

On 30 October 2026 Google, Microsoft and Anthropic moved on one shared robots.txt token, __usage_ai-training. Microsoft reads it as training only. Google plans to read it as a Google-Extended disallow, which also covers grounding in Gemini Apps. What each provider has said, the dates, the wildcard trap, and what a business that wants citations should write.
Technical

PerplexityBot and Perplexity-User: what each one fetches, and what the stealth crawling report means for your robots.txt

Perplexity documents two user agents. PerplexityBot indexes pages for Perplexity search and obeys robots.txt. Perplexity-User fetches a page when someone asks a question, and Perplexity's own docs say it generally ignores robots.txt. In August 2025 Cloudflare reported an undeclared crawler it tied to Perplexity, and Perplexity denied it. The two user-agent strings, what each one fetches, what the report claimed, and the robots.txt and firewall rules a business site should ship.
Technical

GPTBot, OAI-SearchBot and ChatGPT-User: what each OpenAI bot does, and what to put in robots.txt

OpenAI runs three bots most sites see: GPTBot crawls for model training, OAI-SearchBot decides whether you appear in ChatGPT search, and ChatGPT-User fetches a page when a person asks. The exact user-agent strings, how robots.txt applies to each, the four published IP lists we checked on 9 October 2026, and three copy-paste robots.txt files for the three common choices.
Technical

Do AI crawlers respect robots.txt? What each company promises, and what the logs show

OpenAI, Anthropic and Perplexity all say their crawlers obey robots.txt. OpenAI and Perplexity both carve out the fetches a user triggers. Cloudflare's own tests caught one company crawling under a disguised browser identity, and HasData found 234 of 592 sites that ban GPTBot in robots.txt still served it a page. What each company promises, what the evidence shows, whether AI bots are blocked by default, and the layered way to block them.
Technical

ClaudeBot, Claude-User and Claude-SearchBot: what robots.txt does to each of Anthropic's crawlers

Anthropic runs three bots: ClaudeBot for training data, Claude-SearchBot for search indexing and Claude-User for fetches a person asks for. Its help page, re-read on 6 November 2026, says all three obey robots.txt and Crawl-delay, and it links one IP list for all of them. What a Disallow does to each, how to detect ClaudeBot in your logs, and why Anthropic warns against blocking by IP.
Technical

You can opt out of Google's AI Overviews now. Here is what the switch does and does not touch.

Since 31 August 2026 every site in Search Console has a Search generative AI control that excludes it from AI Overviews, AI Mode and generative AI features in Discover. Google's help page says it is not a ranking signal, does not affect AI training, and takes a few days to apply. On 6 October Google added a note on child properties overriding a parent. What the switch covers, the inheritance trap, and why most businesses should leave it on.
Technical

The September 2026 spam update finished on 8 October. If your GEO pages were built at scale, read this first.

Google's September 2026 spam update ran from 24 September to 8 October, per Google's Search Status Dashboard, against about two days for its previous two spam updates. Google's own AI features page says a page must be eligible for normal Search to be a supporting link in AI Overviews or AI Mode. What that means for GEO programmes that publish hundreds of templated pages, what Google has and has not said, and how to read your own data before blaming the update.
Technical

Lighthouse now audits agentic browsing. What it checks, and what it says about AI answers

Lighthouse 13.5, released on 18 September 2026, adds an Agent Resource Discovery audit and groups it with llms.txt under agent discovery, inside an experimental Agentic Browsing category with no 0 to 100 score. What each audit checks, why Not Applicable is often the right result, the file-name mismatch in the new audit, and what none of it tells you about being cited.
Technical

Speakable and FAQ rich results: two levers that stopped paying

Google retired FAQ rich results on 7 May 2026 and speakable only ever fed a news feature for smart speakers. What to strike from an audit, what to keep for machine reading, and why.
Technical

Is Cloudflare blocking ChatGPT from your website?

The September 2026 Cloudflare change only recommends restrictions for sites that earn money from ads. The change that quietly blocks AI crawlers on a restaurant or clinic site landed in July 2025.
Technical

Does llms.txt do anything? 97 percent of the files Ahrefs found got zero requests

Ahrefs checked 137,210 domains and found almost nothing fetching llms.txt. Google says it neither harms nor helps. Here is the one case John Mueller conceded, and how to check your own logs.
Technical

Block, allow or price: the AI crawler decision after Cloudflare's September flip

Cloudflare's 15 September 2026 default is a publisher story. The decision a normal business site has to make is narrower: which crawlers to block, which to allow, and whether pricing is even available to you.
Technical

If your menu loads in JavaScript, no AI can read it

Why a JavaScript-loaded menu is invisible to ChatGPT, Claude and Perplexity, how to run the disable-JavaScript test, and whether the fix is a template or a rebuild.
Technical

Schema markup for AI search: which types matter

Which schema types are worth deploying for a hospitality business, what to put in each, and the honest limit: a matched study found no lift in AI citations.
Technical

llms.txt: what it is, how to write one, and whether it does anything yet

A plain guide to the llms.txt proposal with a working example, and an honest assessment: Google says it does not use the file and no major provider reads it in answers.
Technical

AI crawlers and robots.txt: which bots to allow and what you give up

GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest: what each is for, and how to allow answering bots while deciding separately on training access.