AI Crawler Checker
Can ChatGPT Read Your Site?
See Which AI Crawlers Are Let In, And What They Receive When They Arrive.
What AI Crawlers Get
What to do:
Crawler By Crawler
What your robots.txt tells each crawler about this page.
| Crawler | What it does | This page |
|---|
Lines To Let AI Search Crawlers In
Paste these at the top of your robots.txt. They let the assistants' search crawlers in and leave your other rules as they are.
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: DuckAssistBot Allow: /
The llms.txt Generator builds the file from your sitemap, with each key page's own title and description.
What To Do Next
Being readable is the first step; being the business the assistant recommends is the work. groas does that for clients: book a consultation and bring this report.
How It Works
- 01
Enter A Page
Your homepage, or the page you most want an assistant to quote. No login is needed.
- 02
We Read It As A Bot
We fetch the page, your robots.txt, your sitemap and llms.txt as a plainly labelled bot, without running JavaScript, the way most AI crawlers do.
- 03
See Who Is Let In
You get a verdict, a table of 18 AI crawlers marked allowed or blocked, and the lines to paste into robots.txt if the wrong ones are kept out.
The AI Crawlers We Test
There are two kinds. Search and answer crawlers read pages so an assistant can answer with them: block these and you cannot be cited. Training crawlers collect text to train future models: blocking those is your call and does not affect AI search.
| Crawler | Company | Kind | What it does |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search and answers | Builds the index ChatGPT searches |
| ChatGPT-User | OpenAI | Search and answers | Opens a page when a ChatGPT user asks about it |
| Claude-SearchBot | Anthropic | Search and answers | Builds the index Claude searches |
| Claude-User | Anthropic | Search and answers | Opens a page when a Claude user asks about it |
| PerplexityBot | Perplexity | Search and answers | Builds the index Perplexity answers from |
| Perplexity-User | Perplexity | Search and answers | Opens a page when a Perplexity user asks about it |
| Googlebot | Search and answers | Google Search, AI Overviews and AI Mode | |
| Bingbot | Microsoft | Search and answers | Bing Search and Microsoft Copilot |
| Applebot | Apple | Search and answers | Siri and Spotlight suggestions |
| DuckAssistBot | DuckDuckGo | Search and answers | DuckDuckGo’s AI-assisted answers |
| GPTBot | OpenAI | Training | Collects text to train OpenAI’s models |
| ClaudeBot | Anthropic | Training | Collects text to train Anthropic’s models |
| Google-Extended | Training | Controls use of your pages to train Gemini | |
| Applebot-Extended | Apple | Training | Controls use of your pages to train Apple’s models |
| Meta-ExternalAgent | Meta | Training | Collects text for Meta’s AI models |
| Amazonbot | Amazon | Training | Collects pages for Alexa and Amazon’s AI models |
| CCBot | Common Crawl | Training | A public web archive many AI models are trained on |
| Bytespider | ByteDance | Training | Collects text for ByteDance’s models |
Robots.txt Lines That Let AI Search In
Paste these at the top of your robots.txt. They let the assistants’ search crawlers fetch every page and leave your rules for other crawlers as they are.
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: DuckAssistBot Allow: /
An Example Report
An invented plumber’s site with the two faults we see most: a robots.txt that shuts out the assistants’ crawlers, and a page whose text is added by JavaScript. The verdict: “AI assistants are being kept away from this page”. Once a page passes, the AI Visibility Checker shows whether the assistants recommend it.
robots.txt blocks 3 AI search crawlers
FixOAI-SearchBot, ChatGPT-User and PerplexityBot may not fetch this page. These crawlers read pages to answer questions; blocked, the assistant cannot read or cite you.
What to do: Remove the Disallow rule that covers them, or paste the Allow lines shown on this page at the top of your robots.txt.
Almost no text in the HTML (1 word)
FixMost AI crawlers do not run JavaScript: what they read is the raw HTML. If your page shows more than this in a browser, the rest is added by scripts and the crawlers never see it.
What to do: Render the page on the server or prerender it, so the headings and copy are in the HTML itself.
The page is missing a meta description and an h1 heading
CheckThese tell a machine what the page is about before it reads the body.
What to do: Give every page one title, one meta description and exactly one h1 that say what the page offers.
No structured data (JSON-LD)
CheckStructured data states who you are and what you sell in a form machines read without guessing.
What to do: Add Organization markup site-wide and, where it fits the page, LocalBusiness, Product or FAQPage.
The page loads for a bot
OKWe asked for the page as a plainly labelled bot and received it (HTTP 200).
What to do:
No noindex or nosnippet rule
OKNothing on the page asks search engines to leave it out or to not quote it.
What to do:
A sitemap was found
OKCrawlers can discover your pages from it.
What to do:
2 training crawlers are blocked
NoteBlocked: GPTBot and ClaudeBot. Training crawlers collect text to train future models. Allowing or blocking them is your decision, and blocking them does not take you out of AI search answers.
What to do:
No llms.txt (optional)
Notellms.txt is a proposed file that lists a site’s key pages for AI tools. No major assistant has said it uses the file to choose answers, so treat it as cheap and optional.
What to do:
FAQ
Which crawlers matter, what to block and what to leave open.
How do I know if ChatGPT can read my website?
Three things have to be true. Your robots.txt must not block OpenAI’s crawlers (OAI-SearchBot for its search index and ChatGPT-User for pages a user asks about). Your firewall or CDN must not turn bots away. And the page’s text must be in its HTML, because these crawlers do not run JavaScript. Enter your page above and the tool checks all three.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects text to train OpenAI’s models. OAI-SearchBot builds the index ChatGPT searches when it answers with sources, and ChatGPT-User fetches a page when a user asks about it. Blocking GPTBot keeps your text out of training; blocking the other two keeps you out of ChatGPT’s answers. Anthropic and Google split their crawlers the same way.
Should I block AI crawlers?
Block training crawlers if you do not want your text used to train models: that is a fair choice and costs you nothing in search. Do not block the search and answer crawlers unless you want to be absent from AI answers, because an assistant cannot recommend or cite a page it may not read.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is the training crawler. ChatGPT’s search answers come from OAI-SearchBot and from pages ChatGPT-User fetches, so a site that blocks only GPTBot can still be cited.
Why does it matter whether my content is in the HTML?
Most AI crawlers download the HTML and stop. They do not run the JavaScript that builds a page in a browser. If your headings and copy are added by scripts, the crawler receives a nearly empty page, however good it looks to a visitor. Server rendering or prerendering fixes it.
Do I need an llms.txt file?
It is optional. llms.txt is a proposed file that lists a site’s key pages for AI tools. No major assistant has said it uses the file to choose its answers, so it will not fix a blocked or empty page. It is cheap to add, and the llms.txt Generator builds one from your sitemap.
The report says crawlers are allowed. Why am I still not in AI answers?
Being readable only makes you eligible. Whether an assistant names you depends on what the web says about you: reviews, mentions, comparisons and how clearly your own pages answer the question. The AI Visibility Checker shows whether the assistants recommend you today.
Can this tool see my firewall rules?
Only indirectly. It asks for your page as a plainly labelled bot; if it is refused, bots in general are being turned away. It cannot see a rule that blocks one crawler by name or by address. Your CDN’s bot or AI-crawler settings show that.
Readable Is Step One. Recommended Is The Goal.
Book a consultation: groas works on what AI assistants and search engines say about your business, alongside your ads, for a fixed monthly fee. 7-day money-back guarantee.