If I allow the search bots, will my content be used for training?

Not by the same bot. OpenAI, Perplexity and Anthropic each describe separate bots for search and for training. Allowing the search bot is a different choice from allowing the training bot.

Does Google-Extended control AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. For how your pages appear in AI features in Search, Google points to standard controls such as noindex and snippet settings.

Do user-triggered bots follow robots.txt?

Often not. OpenAI says robots rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page. They do not decide whether you appear in search.

Do I need an llms.txt file for AI bots to read my site?

Google says no special AI files or markup are needed to appear in its AI features. The bot documentation reviewed here does not mention one, so fix access first.

What does Nova do that the skill file doesn't?

The skill file gives any agent the method. Nova reads your live robots.txt, tests each bot, drafts the change for approval and re-checks after the wait.

AUD-01 · Audit Maneuver

Why can't ChatGPT or Perplexity see my website?

Usually a robots.txt rule or a firewall setting is blocking the bot that feeds that engine's search. OpenAI, Perplexity and Anthropic each run separate bots for search and for training, so you can allow the search bots and still opt out of training. Allow the search bots, then give the change about a day to take effect.

Take it to Nova Read the Maneuver

Free to read, no signup. Nova checks every AI bot against your live robots.txt and reports what is blocked.

01 · What the AI companies say

Each AI company publishes which bots it runs

Straight from their own documentation. Search bots and training bots are separate, so you can choose each.

OpenAI

Block OAI-SearchBot and you leave ChatGPT search

OAI-SearchBot surfaces sites in ChatGPT search, and blocking it keeps a site out of those answers. GPTBot is a separate bot for training, and robots.txt changes take about 24 hours.

Overview of OpenAI crawlers

Perplexity

Allow PerplexityBot to be listed

PerplexityBot indexes sites for Perplexity search and is not used for training. Perplexity says to allow it by user agent and by its published IP ranges, and to check both, not just the user agent.

Perplexity crawlers

Anthropic

Three Claude bots, three jobs

ClaudeBot is for training, Claude-User fetches pages when someone asks Claude a question, and Claude-SearchBot supports search. Anthropic advises against blocking by IP address, which can stop the bots reading your robots.txt.

Does Anthropic crawl data from the web?

Google

Google-Extended does not touch Search

Google-Extended is a robots.txt token for Gemini training and grounding. Google says it does not affect a site's inclusion in Google Search, so it is not the switch for AI Overviews.

Google's common crawlers

02 · The Maneuver

Five steps to clear the way for AI bots

Do it by hand with these steps, or hand the whole thing to Nova.

Read your robots.txt
Open yourbrand.com/robots.txt. Look for blanket Disallow rules and for any rule naming OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Claude-SearchBot or Claude-User.
Nova: fetches your robots.txt and lists every rule that touches an AI bot.
Separate search bots from training bots
Decide each on its own. Allowing OAI-SearchBot, PerplexityBot and Claude-SearchBot lets engines show you in answers; blocking GPTBot or ClaudeBot opts you out of training.
Nova: sorts each bot into search, training or user-triggered, and shows what each rule does.
Check your firewall and CDN
A security rule can block a bot that robots.txt allows. Allow search bots by user agent and by their published IP ranges, and avoid blocking by IP alone.
Nova: flags bots that robots.txt allows but a firewall is likely to block.
Make the change
Edit robots.txt at the top level of each domain and subdomain you want covered. Keep a copy of the old file.
Nova: drafts the exact robots.txt lines for your approval. Nothing is changed for you.
Wait, then check again
OpenAI says robots.txt changes take about 24 hours. Re-test after a day, then confirm the bots show up in your logs.
Nova: re-checks after a day and reports which bots are now allowed.

Nova

Short on time? Nova, your AI brand visibility expert, runs this Maneuver on your brand and tracks the result.

Take it to Nova

Bonus

Free AI crawler checker. Get a quick check before you start.

Try it free

03 · What Nova hands back

Every AI bot, its rule and the fix

Sample output for a fictional site
Bot	What it does	Rule found	Effect	Fix
OAI-SearchBot	ChatGPT search	Disallow: /	Site missing from ChatGPT search	Allow in robots.txt
PerplexityBot	Perplexity search	Allowed, blocked by firewall	Cannot reach pages	Allow by user agent + IP ranges
GPTBot	Training	Disallow: /	Opted out of training	No change, your choice

Nova then re-checks your robots.txt a day after the change, so you see which bots now reach your site.

04 · Free skill file

Take this Maneuver into your own AI agent

Get crawler-clearance.md, a ready-to-use skill file with the bot list, the questions to ask about your setup and the robots.txt patterns to review. Drop it into Claude, Cursor or any agent that reads SKILL.md files.

crawler-clearance.md

Skill file for Claude, Cursor and other agents

Search, training and user-triggered bots, sorted
A review checklist for robots.txt and firewalls
A re-test routine for after the change

05 · FAQ

Questions, answered.

06 · Related Maneuvers

AUD-02ChatGPT is saying the wrong things about my brand AUD-03AI never cites my best page GAP-02What is query fan-out and how do I win it?

07 · Relevant reads

Relevant reads

All articles

AI brand visibility

What is AI search visibility, and how do you measure it?

Oct 9, 20263 min read

Let Nova clear the way for AI bots

Nova opens with this Maneuver loaded. You approve every change before it ships.