Free AI bot checker

See which of 12 AI crawlers, from GPTBot to ClaudeBot, your robots.txt lets in.

FreeLive robots.txt12 bots

Rules read as in RFC 9309

Paste the contents of yourdomain.com/robots.txt

Checked in your browser. Nothing is sent.

— / 12Enter a domain or paste a robots.txt

Searchdecides if you can be cited

OAI-SearchBotChatGPT search—Waiting

PerplexityBotPerplexity search—Waiting

Claude-SearchBotClaude search—Waiting

GooglebotGoogle Search, AI Overviews and AI Mode—Waiting

MSBingbotBing and Copilot—Waiting

On requestopens a page a user asked about

ChatGPT-UserChatGPT opening a page for a user—Waiting

Perplexity-UserPerplexity opening a page for a user—Waiting

Claude-UserClaude opening a page for a user—Waiting

Trainingdecides if models learn from you

GPTBotOpenAI model training—Waiting

ClaudeBotAnthropic model training—Waiting

Google-ExtendedGemini training and grounding—Waiting

CCCCBotCommon Crawl, used to train many models—Waiting

Three kinds of bot

Not every AI bot trains.

Search bots decide if you can be cited. Training bots only decide what models learn.

OAI-SearchBot

PerplexityBot

Claude-SearchBot

Search

Finds pages to show and cite in AI answers. Keep these allowed.

Sources: OpenAI, Perplexity, Anthropic

ChatGPT-User

Perplexity-User

Claude-User

On request

Opens a page because a user asked. OpenAI and Perplexity say robots.txt may not apply.

Sources: OpenAI, Perplexity

GPTBot

ClaudeBot

Google-Extended

Training

Decides if models learn from you. Google says Google-Extended doesn't affect Search.

Sources: OpenAI, Google

Vendor documentation checked 29 Oct 2026.

robots.txt for AI

Stay citable, your way.

Search and training use separate bots, so you can allow one and block the other.

robots.txt · example

# Let AI search find and cite youUser-agent: OAI-SearchBotUser-agent: PerplexityBotUser-agent: Claude-SearchBotAllow: / # Optional: opt out of model trainingUser-agent: GPTBotUser-agent: ClaudeBotUser-agent: Google-ExtendedUser-agent: CCBotDisallow: / User-agent: *Allow: /

The most specific group wins

A bot follows the group naming it, else the * group.

The longest rule wins

Inside that group, the longest matching path decides.

Allow wins a tie

Equal length? Allow beats Disallow. No file means all allowed.

How rules are read: RFC 9309, Robots Exclusion Protocol. Training opt-out is a choice, not a recommendation.

FAQ

AI bots, answered.

Which AI bots does this check?

OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, ClaudeBot, Googlebot, Google-Extended, Bingbot and CCBot.

Should I block GPTBot?

It is your call. GPTBot is OpenAI's training bot. ChatGPT search uses OAI-SearchBot, a separate bot you can keep allowed.

Does blocking Google-Extended hide me from AI Overviews?

Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are controlled through Googlebot.

How does the domain check work?

We read your live robots.txt from our server and judge each bot with the same rules the AY Rank app uses. Results are kept for 7 days. You can also paste a file to test changes before you publish them.

Is my site cited if all bots are allowed?

Not necessarily. Access is required, but engines still choose which pages to quote. Track that with the AI visibility checker.

Does the paste check send my file anywhere?

No. The file is read in your browser with the same rules the AY Rank app uses (RFC 9309).

Can AI bots read my page if robots.txt allows them?

Only if the text is in the HTML. Most AI crawlers don't run JavaScript. See what they get with What AI sees.