Retrievability and crawler access
The mechanical prerequisites. If one of these is wrong, nothing else you spend money on can work, because the page is never read in the first place.
Short answer
An AI crawler fetches your page once, reads the HTML that comes back, and runs no JavaScript. If your content is assembled in the browser, or your CDN turns the answering bots away, or the URL it has on file returns a 404, you are absent from the answer regardless of how good your business is. This is the cheapest and highest-confidence work in the whole discipline.
No major AI crawler executes JavaScript
Across an analysis of more than 500 million crawler fetches by Vercel and MERJ, GPTBot, ClaudeBot, PerplexityBot and the rest issued a request and read the returned HTML without running a single script. They sometimes download JavaScript files and never execute them.
The exceptions are Google's AI surfaces, which inherit Googlebot, Microsoft Copilot, which inherits Bingbot, and AppleBot. Everything else sees the raw response.
The test takes ten seconds. Disable JavaScript in your browser, reload your homepage, and look at what survives. If your hours, your address and your services vanish, that is what ChatGPT, Claude and Perplexity see.
Answering crawlers and training crawlers are different decisions
This is the most expensive misunderstanding in the field. Two separate groups of bots get treated as one.
- Training crawlers build a model's background knowledge:
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended. Blocking these is a legitimate content-licensing decision. - Answering crawlers fetch pages so you can be cited in a live answer:
OAI-SearchBot,Claude-SearchBot,PerplexityBot,Bingbot. Blocking one of these removes you from that surface completely. - User-initiated fetchers fire when a person asks about you by name:
ChatGPT-User,Claude-User,Perplexity-User. Two of them document that they ignore robots.txt, so a rule here is a statement of intent rather than a control. Google-Extendedis the trap. It governs Gemini training and grounding in some Google products. It does not affect AI Overviews or AI Mode eligibility. People block it believing they are protecting something and then wonder why nothing changed.
You can allow answering while blocking training. Most sites that think they have done this have done the opposite.
Your CDN is a second robots.txt
Cloudflare has blocked AI crawlers by default since 1 July 2025. A site can have a perfectly permissive robots.txt and still be refusing every answering bot at the edge, with nothing in the site's own configuration to show for it. We check both. Ours is public at /robots.txt if you want to see what a deliberate one looks like.
Dead URLs cost more than they do in search
AI crawlers are unusually bad at following a site's history. The same Vercel dataset put 404 rates at 34.8 percent for ChatGPT's crawler and 34.2 percent for Claude's, against 8.2 percent for Googlebot. The gap is stale URLs and redirect chains that Google has long since re-learned.
It is unglamorous and it is the highest-confidence fix available.
What about schema?
We deploy it, and we will not sell it to you as a visibility lever. A matched study of 1,885 pages that added JSON-LD against roughly 4,000 controls found no major uplift on any platform, and Google states that no special structured data is needed for AI Overviews or AI Mode. Schema earns conventional rich results and helps disambiguate your business from a similarly named one. Those are real. Citations are not what it buys.
What you get
- A crawler access report covering robots.txt, CDN rules and every answering-bot token, with the exact lines quoted.
- Server-render verification: the same pages fetched the way the crawlers fetch them, with JavaScript disabled, and a list of what disappears.
- A cleared 404 and redirect-chain list.
- Schema deployed correctly, described honestly.
Common questions
Usually not. Most of this is server configuration, robots.txt, CDN rules and dead URLs. If your site assembles its content in the browser, that part does have to change, because the AI crawlers cannot see it. We tell you upfront if that is the case rather than discovering it halfway through.
No, and this is the misunderstanding that costs the most. GPTBot is the training crawler. OAI-SearchBot is the one that surfaces sites in ChatGPT search. You can block GPTBot and still appear in answers. Block OAI-SearchBot and you disappear from that surface entirely.
You request the page using the crawler user-agent string and see what comes back. A Cloudflare managed rule will typically return a challenge or a 403 rather than your page. We run this as part of the access report, and it is a common finding because the block was a platform default rather than anyone’s decision.
What else an engagement covers
Want to know whether this is your problem?
Book a 30 minute call. We will fetch your site the way the crawlers do and tell you which of the prerequisites is failing. If none of them are, we will say so.