Topic

The technical work that decides whether an AI engine can read you

Crawler access, server-rendered HTML, schema and llms.txt: which technical work an answer engine needs, and which of it does nothing at all.

Short answer

Two conditions decide most of it. The answering crawlers have to get through your CDN and your robots.txt, which means allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot, and the content they came for has to exist in the raw HTML rather than being assembled by JavaScript in the browser. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are a separate and lower-stakes decision, and blocking the wrong one removes you from answers while protecting nothing. Schema is worth having because it states filterable attributes as data, though Ahrefs' matched study of 1,885 pages that added JSON-LD against 4,000 controls found no lift in AI citations, and llms.txt is close to inert for discovery, with Ahrefs finding 97 percent of the published files it located across 137,210 domains had received zero requests.

The technical half of this work is unglamorous, cheap and finite, and it is also the half that most often explains why a business is missing from AI answers entirely. A menu or a price list that only exists after JavaScript runs is invisible to the crawlers behind ChatGPT, Claude and Perplexity, and that single template decision outweighs every content tactic sold on top of it. NovaTechRay puts the crawler and rendering checks first because they are binary: either the page can be read or nothing else you do matters. The part we will concede, which most vendors will not, is that two of the most-sold technical deliverables in this field, schema as a citation lever and llms.txt, have evidence pointing at roughly nothing. Ship them if you want them for other reasons, and do not let anyone bill you for them as the reason you get cited.

In this topic

Technical

Is Cloudflare blocking ChatGPT from your website?

The September 2026 Cloudflare default only touches pages that display ads. The change that quietly blocks AI crawlers on a restaurant or clinic site landed in July 2025.
Technical

Does llms.txt do anything? 97 percent of the files Ahrefs found got zero requests

Ahrefs checked 137,210 domains and found almost nothing fetching llms.txt. Google says it neither harms nor helps. Here is the one case John Mueller conceded, and how to check your own logs.
Technical

Block, allow or price: the AI crawler decision after Cloudflare's September flip

Cloudflare's 15 September 2026 default is a publisher story. The decision a normal business site has to make is narrower: which crawlers to block, which to allow, and whether pricing is even available to you.
Technical

If your menu loads in JavaScript, no AI can read it

Why a JavaScript-loaded menu is invisible to ChatGPT, Claude and Perplexity, how to run the disable-JavaScript test, and whether the fix is a template or a rebuild.
Technical

Schema markup for AI search: which types matter

Which schema types are worth deploying for a hospitality business, what to put in each, and the honest limit: a matched study found no lift in AI citations.
Technical

llms.txt: what it is, how to write one, and whether it does anything yet

A plain guide to the llms.txt proposal with a working example, and an honest assessment: Google says it does not use the file and no major provider reads it in answers.
Technical

AI crawlers and robots.txt: which bots to allow and what you give up

GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest: what each is for, and how to allow answering bots while deciding separately on training access.