# NovaTechRay — robots.txt # Reviewed 2026-09-10. Token spellings taken from operator docs: # OpenAI https://developers.openai.com/api/docs/bots # Anthropic https://support.claude.com/en/articles/8896518 # Perplexity https://docs.perplexity.ai/guides/bots User-agent: * Allow: / # --------------------------------------------------------------- # ANSWERING BOTS — these decide whether you appear inside an AI # answer. Blocking any one removes you from that surface entirely. # Keep every one of these allowed. # --------------------------------------------------------------- User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Bingbot Allow: / User-agent: Googlebot Allow: / # --------------------------------------------------------------- # USER-INITIATED FETCHERS — fired when a person asks about you by # name. ChatGPT-User and Perplexity-User document that they ignore # robots.txt, so these lines are a statement of intent, not a control. # --------------------------------------------------------------- User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / # --------------------------------------------------------------- # TRAINING CRAWLERS — a separate decision from the above. Allowed # here deliberately: inclusion in model training is how a small brand # becomes known to a model without a live retrieval step. # To opt out later, change Allow to Disallow in this block only. # Note: Google-Extended and Applebot-Extended are robots.txt control # tokens with no user-agent string, and Google-Extended does NOT # affect AI Overviews or AI Mode eligibility. # --------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: CCBot Allow: / Sitemap: https://www.novatechray.com/sitemap.xml