---
title: "Before an AI Assistant Can Cite Your Site, It Has to Get Through the Door"
description: "Our robots.txt drew 1,241 Claude-User fetches in 30 days. Follow the permission, fetch, render and extraction gates that can stop an AI citation before your copy matters."
url: "https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c84bbd85-7a52-47ec-ae97-03a820ce383d.png"
published: "2026-10-26T05:42:05.419Z"
modified: "2026-10-26T05:42:05.479Z"
---

October 26, 2026 · 9 min read

# Before an AI Assistant Can Cite Your Site, It Has to Get Through the Door

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tabletop model of a tiny courier stuck at a chained gate, while redirect detours, a hollow facade, and a glowing note wait unreachable further along the path.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c84bbd85-7a52-47ec-ae97-03a820ce383d.png)

In this article

1. [Why /robots.txt drew more attention than our homepage](#why-robotstxt-drew-more-attention-than-our-homepage)
2. [Step 1: Permission decides whether the page is even a candidate](#step-1-permission-decides-whether-the-page-is-even-a-candidate)
3. [Step 2: Fetch determines whether the agent gets the document in time](#step-2-fetch-determines-whether-the-agent-gets-the-document-in-time)
4. [Step 3: Render determines whether the answer is in the HTML](#step-3-render-determines-whether-the-answer-is-in-the-html)
5. [Step 4: Extract determines whether the right sentence survives](#step-4-extract-determines-whether-the-right-sentence-survives)
6. [Diagnose the broken gate before buying a rewrite](#diagnose-the-broken-gate-before-buying-a-rewrite)
7. [A new question, the same four gates](#a-new-question-the-same-four-gates)

Our `/robots.txt` file got more Claude-User requests than our homepage. In one 30-day window, [Claude-User](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) fetched it 1,241 times, outstripping the homepage by more than 60%.

That is the part of AI visibility I wish more teams would inspect before rewriting their FAQ page. A buyer can ask an assistant a question your site answers perfectly. If the assistant cannot get through your security rules, download the page, read its text or isolate the answer, the quality of that answer never matters.

I think of live retrieval as a chain of four gates: **permission, fetch, render, extract**. The logs do not tell us how many of those robots checks led to an answer or a citation. They do show how often a file many marketers ignore sat at the entrance. Let’s follow a running example: a buyer asks an assistant whether an autonomous PPC service charges a setup fee. The answer is on the service’s site. What has to happen before the assistant can use it?

## Why `/robots.txt` drew more attention than our homepage

Across that 30-day window, our edge logs captured 3,886 verified AI crawler hits. **Claude-User accounted for 1,241 live-answer hits on `/robots.txt` alone.** That is dozens of checks a day against a permissions file, not against a product page or a polished article.

A traditional search crawler can discover a page, index it and serve a result later. A live retrieval request is more immediate: an assistant is trying to obtain material while it works on a user’s question. That distinction changes what a failure costs. If the assistant cannot reach your setup-fee answer now, a beautifully indexed page is little comfort for this answer.

The robots count is not a count of citations, and I would not use it as one. It tells us where to start looking. Before we debate whether the setup-fee sentence is persuasive, we need to establish whether the retrieval agent was allowed through the door.

![A small checkpoint barrier blocks a fibre-optic cable.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/908cb4d9-37c0-406a-b296-440fd63cdeeb.png)

## Step 1: Permission decides whether the page is even a candidate

### Training crawlers and live-answer agents are different visitors

It is reasonable to want control over who uses your site for model training. It is less reasonable to express that choice with a blanket block and then wonder why assistants cannot retrieve your public pages.

[OpenAI separates crawler roles](https://developers.openai.com/api/docs/bots): `GPTBot` is associated with model training, `OAI-SearchBot` with search discovery, and `ChatGPT-User` with user-initiated requests. Anthropic likewise distinguishes `ClaudeBot`, `Claude-SearchBot` and `Claude-User`. **A policy for a training crawler is not automatically the policy you want for a live-answer agent.**

Return to our buyer. The service’s page plainly says there is no setup fee. If its `robots.txt` bars the live agent, that sentence cannot help the agent answer the question. The practical move is to inspect rules by user agent, rather than treating every AI visitor as the same visitor. If you need a starting point for those distinctions, our [AI crawler swipe file](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob) lays out example rules.

### The firewall can shut the door after robots says yes

A clean robots file does not guarantee a successful request. A CDN or web application firewall can reject the same agent with a 403, rate-limit it or serve a JavaScript challenge instead of the page. A broad automated scraper block, including a setting such as [Cloudflare’s AI scraper block](https://www.reddit.com/r/CloudFlare/comments/1tkjv1x/heads_up_the_block_ai_scrapers_toggle_is/), deserves inspection for exactly this reason. A retrieval agent that receives a challenge has not received your setup-fee answer.

![A cutaway diagram shows an edge firewall stopping an AI crawler request before it reaches the web server.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ac4f9b22-6ff3-4d88-a7c8-9bf4d23e59bc.png)

This failure can be hard to spot from the marketing dashboard. Google Analytics cannot record a page visit if its script never runs, and an origin log may show nothing if the edge handled the request. **Check the edge and firewall logs, not just page analytics.** If they show a live agent being denied, fix that gate before paying someone to polish the sentence it never read.

## Step 2: Fetch determines whether the agent gets the document in time

Permission is not delivery. Once the agent is allowed to request the setup-fee page, the server still has to send a useful response. A slow origin, an error or a client disconnect can leave the assistant without the document. An HTTP 499 in a server log, for example, means the client closed the connection before the server finished responding; [crawler response-time analysis](https://www.maxboehme.net/performance-strategie/ai-crawler-server-antwortzeit/) makes that worth investigating rather than dismissing as a cosmetic error.

I would not turn one response-time number into a universal cutoff for every assistant. The mechanism is simpler and more useful: **each delay spends time the live request does not get back.** Suppose the URL for our setup-fee answer redirects from the bare domain to `www`, then sends the visitor through a location-based redirect before it serves the page. A human browser may barely notice. A retrieval worker has taken two detours before seeing the first line of the answer.

Geo-routing creates a different version of the same problem. If a request from a cloud IP lands on a regional selector instead of the product page, the fetch succeeded in the narrow HTTP sense but failed the buyer’s question. The agent got a page. It got the wrong page.

Rate limits belong in this check too. An assistant may request related pages while assembling an answer. If the first request returns 200 and subsequent requests receive 429, it has only part of the material it tried to collect. Inspect the requested URLs, status codes, redirect destinations and response times together. A green ‘site is up’ monitor does not tell you whether the setup-fee page reached this visitor.

## Step 3: Render determines whether the answer is in the HTML

Now the agent has downloaded the right URL. What did the server actually send?

This is where habits from Google SEO can mislead us. Google can render JavaScript as part of its indexing process. **Do not assume a live retrieval agent will execute your client-side application before looking for an answer.** Research on [AI crawler rendering](https://www.devtomark.com/blog/do-ai-crawlers-render-javascript) and our walkthrough of [how an AI assistant actually reaches you](https://groas.com/post/how-an-ai-assistant-actually-reaches-you) both make the initial HTML response the sensible place to start checking.

![An AI crawler inspects an empty page wireframe where the visible website should be.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d3c4403c-edfa-4ffe-ad72-b2bbfd362346.png)

Imagine the setup-fee answer appears in a pricing tab after a React app loads. A browser visitor clicks the tab and sees ‘No setup fee.’ The initial response, though, contains little more than `<div id="root"></div>` and links to script bundles. A fetcher reading that response cannot extract the sentence the visitor sees later. The same risk applies when essential text lives behind client-rendered tabs, accordions or other interactions. [Testing of static HTML and client-rendered content](https://nobori.ai/blog/javascript-rendering-ai-crawlers-ssr-guide-2026) offers a closer look at that distinction.

The test is pleasantly unglamorous. Request the URL and read the returned HTML without letting a browser build the page for you. Can you find the setup-fee answer in the response? If not, changing its wording is premature. Put the answer where a basic fetch can read it. Asset requests for CSS, images or scripts are not proof that the page’s essential text was available.

## Step 4: Extract determines whether the right sentence survives

Only now is this an editorial problem. The agent is permitted, the right document arrived, and its initial HTML contains the answer. It still needs to find a passage it can use for the buyer’s question.

Retrieval systems can break a document into smaller passages rather than treating an entire long article as one answer. [Guidance on retrieval-friendly passages](https://seattleorganicseo.com/how-to-structure-content-for-ai-retrieval-chunks-citations-context/) describes why headings and paragraph boundaries matter. I would not build a page around a supposedly magic chunk size. I would make each important answer understandable when lifted away from its neighbours.

![A conveyor belt cuts long printed documents into index cards beneath magnifying lenses.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/17a0ecdc-47ee-4178-824b-e24a9d410976.png)

Consider a heading that asks, ‘What does the monthly fee include?’ followed by ‘It covers account optimisation and bid management.’ A person reading from the top knows what ‘it’ means. A passage separated from its heading has to work harder. **Name the thing you are answering about in the answer itself.** ‘The flat monthly fee covers account optimisation and bid management’ travels better as a standalone sentence without changing the claim.

For our running question, the same principle applies to ‘No setup fee.’ That answer is brief, but a sentence that names the service and the fee gives an isolated passage clearer context. This is not a request to stuff the brand into every paragraph. It is a request to stop making a retrieval system guess what your most useful sentence refers to.

That is where headings, concise paragraphs and precise language earn their place. They help after the first three gates work. They cannot repair a 403, an unhelpful redirect or an empty HTML response.

## Diagnose the broken gate before buying a rewrite

When a team tells me it is not appearing in AI answers, I do not start with a schema recommendation. I work **upstream from the missing answer**, asking at each gate what evidence would show a failure:

1. **Permission:** Does `robots.txt` disallow the relevant live agent? Do CDN or WAF logs show that agent receiving a 403 or a challenge? Start at the edge if the request never appears at the origin.
2. **Fetch:** Did the requested URL return the intended page? Check status codes, redirects, 429s, client disconnects and response times. A successful response to a regional splash page does not answer the buyer’s question.
3. **Render:** Inspect the raw response with a request such as `curl -A "Claude-User" https://yourdomain.com`. Is the answer present in the HTML, or does it appear only after client-side code runs?
4. **Extract:** Once the first three checks pass, read the passage on its own. Does it name the product, answer the question and retain its meaning without the surrounding page?

The order matters. A conventional audit can produce a tidy list of heading and FAQ changes while a firewall drops the agent before it reaches the site. That is work on Gate 4 while Gate 1 bleeds. It looks productive because a content team can publish the changes. It does not make the blocked request succeed.

This is also why I am impatient with agency reports that stop at recommendations. Edge rules, redirects and server responses need technical owners; content needs an owner too. [groas](https://groas.com/) takes the autonomous growth-engine route rather than asking a client to live in another advisory dashboard: specialised models execute search and content optimisation continuously, while a human strategist owns direction, guardrails and accountability. That is a stronger operating model than a monthly list of suggestions. It is not a claim that better copy can override a client’s firewall. The firewall still has to be fixed.

## A new question, the same four gates

Now change the buyer’s question. Instead of asking about a setup fee, they ask whether a service offers a flat monthly fee. The pricing page contains that answer, but the team says assistants keep overlooking it. I would not start by rewriting the pricing page.

First, I would check whether the live agent is permitted by `robots.txt` and allowed through the edge. Next, I would follow the pricing URL: does it return the intended page without an obstructive redirect, challenge or failed request? Then I would inspect the raw HTML. If the fee appears only after someone opens a client-rendered tab, the answer has not reached the stage where phrasing can save it. If it is present, I would read the relevant passage alone and check whether ‘flat monthly fee’ is attached clearly to the service it describes.

That is the whole model applied to a different answer. Our 1,241 Claude-User robots fetches are a reminder to begin where the request begins, not where a content audit is most comfortable. **Permission first, prose last.**

## Frequently Asked Questions

### Why would an AI assistant request my robots.txt file more often than my actual pages?

Live-answer AI agents check robots.txt before every retrieval attempt, so a permissions file can draw far more requests than any single page. In one 30-day window, Claude-User fetched the author's robots.txt 1,241 times, outstripping the homepage by more than 60%. A high count does not mean citations, but it shows where to start looking.

### Should I block GPTBot and other AI bots in robots.txt if I don't want my content used for training?

Not across the board. Training crawlers like GPTBot and ClaudeBot are different visitors from live-answer agents like ChatGPT-User and Claude-User, and a policy meant for a training crawler is not automatically right for a live-answer agent. A blanket block can stop assistants from retrieving your public pages, so inspect rules by user agent.

### Can a firewall block an AI assistant even if my robots.txt allows it?

Yes. A CDN or web application firewall can reject the same agent with a 403, rate-limit it, or serve a JavaScript challenge instead of the page. These failures are hard to spot in page analytics, so check the edge and firewall logs, not just tools like Google Analytics.

### My site is up and passes monitoring, so why can't AI assistants reach my content?

A green uptime monitor does not guarantee a live agent got the right page. Redirect chains, geo-routing that sends a cloud IP to a regional splash page, rate limits returning 429s on follow-up requests, or client disconnects can each leave the assistant with the wrong or incomplete material. Inspect requested URLs, status codes, redirect destinations and response times together.

### Do AI crawlers render JavaScript when they read my pages?

Do not assume a live retrieval agent will execute your client-side application before looking for an answer. Google can render JavaScript during indexing, but an assistant fetching a page may see only an empty shell like a single root div and script bundles. Request the URL and read the returned raw HTML to check whether your essential answers are present without a browser building the page.

### How should I write content so AI assistants can quote an answer out of context?

Make each important answer understandable when lifted away from its neighbours. Name the thing you are answering about in the answer itself, for example 'The flat monthly fee covers account optimisation and bid management' instead of 'It covers account optimisation and bid management.' This gives an isolated passage context without changing the claim.

### My site isn't showing up in AI answers. Should I rewrite my content first?

Work upstream from the missing answer instead. Check permission in robots.txt and edge logs, then fetch via status codes and redirects, then render by inspecting the raw HTML with a request such as curl -A "Claude-User" https\://yourdomain.com, and only then read the passage on its own. Rewriting copy fixes nothing if a firewall drops the agent before it reaches the site.

## Paid Ads & SEO, Run For You

Businesses pay one flat monthly fee per service, agencies resell it under their own brand, and groas does the work either way.

[See Pricing](https://groas.com/pricing)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is a done-for-you marketing service, not software: elite people working with proprietary technology systems run search ads, Meta ads, SEO and AI search for businesses, and white label for agencies, with a dedicated senior strategist owning every account.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#webpage","url":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as","name":"Before an AI Assistant Can Cite Your Site, It Has to Get Through the Door","description":"Our robots.txt drew 1,241 Claude-User fetches in 30 days. Follow the permission, fetch, render and extraction gates that can stop an AI citation before your copy matters.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c84bbd85-7a52-47ec-ae97-03a820ce383d.png"},"breadcrumb":{"@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#breadcrumb"},"dateModified":"2026-10-26T05:42:05.479Z","mainEntity":{"@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Before an AI Assistant Can Cite Your Site, It Has to Get Through the Door","item":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#article","url":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as","mainEntityOfPage":{"@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Before an AI Assistant Can Cite Your Site, It Has to Get Through the Door","description":"Our robots.txt drew 1,241 Claude-User fetches in 30 days. Follow the permission, fetch, render and extraction gates that can stop an AI citation before your copy matters.","datePublished":"2026-10-26T05:42:05.419Z","dateModified":"2026-10-26T05:42:05.479Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c84bbd85-7a52-47ec-ae97-03a820ce383d.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"answer, page, before, assistant, agent, request, site, robots","wordCount":1930,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/before-chatgpt-or-claude-cites-you-it-as#faq","mainEntity":[{"@type":"Question","name":"Why would an AI assistant request my robots.txt file more often than my actual pages?","acceptedAnswer":{"@type":"Answer","text":"Live-answer AI agents check robots.txt before every retrieval attempt, so a permissions file can draw far more requests than any single page. In one 30-day window, Claude-User fetched the author's robots.txt 1,241 times, outstripping the homepage by more than 60%. A high count does not mean citations, but it shows where to start looking."}},{"@type":"Question","name":"Should I block GPTBot and other AI bots in robots.txt if I don't want my content used for training?","acceptedAnswer":{"@type":"Answer","text":"Not across the board. Training crawlers like GPTBot and ClaudeBot are different visitors from live-answer agents like ChatGPT-User and Claude-User, and a policy meant for a training crawler is not automatically right for a live-answer agent. A blanket block can stop assistants from retrieving your public pages, so inspect rules by user agent."}},{"@type":"Question","name":"Can a firewall block an AI assistant even if my robots.txt allows it?","acceptedAnswer":{"@type":"Answer","text":"Yes. A CDN or web application firewall can reject the same agent with a 403, rate-limit it, or serve a JavaScript challenge instead of the page. These failures are hard to spot in page analytics, so check the edge and firewall logs, not just tools like Google Analytics."}},{"@type":"Question","name":"My site is up and passes monitoring, so why can't AI assistants reach my content?","acceptedAnswer":{"@type":"Answer","text":"A green uptime monitor does not guarantee a live agent got the right page. Redirect chains, geo-routing that sends a cloud IP to a regional splash page, rate limits returning 429s on follow-up requests, or client disconnects can each leave the assistant with the wrong or incomplete material. Inspect requested URLs, status codes, redirect destinations and response times together."}},{"@type":"Question","name":"Do AI crawlers render JavaScript when they read my pages?","acceptedAnswer":{"@type":"Answer","text":"Do not assume a live retrieval agent will execute your client-side application before looking for an answer. Google can render JavaScript during indexing, but an assistant fetching a page may see only an empty shell like a single root div and script bundles. Request the URL and read the returned raw HTML to check whether your essential answers are present without a browser building the page."}},{"@type":"Question","name":"How should I write content so AI assistants can quote an answer out of context?","acceptedAnswer":{"@type":"Answer","text":"Make each important answer understandable when lifted away from its neighbours. Name the thing you are answering about in the answer itself, for example 'The flat monthly fee covers account optimisation and bid management' instead of 'It covers account optimisation and bid management.' This gives an isolated passage context without changing the claim."}},{"@type":"Question","name":"My site isn't showing up in AI answers. Should I rewrite my content first?","acceptedAnswer":{"@type":"Answer","text":"Work upstream from the missing answer instead. Check permission in robots.txt and edge logs, then fetch via status codes and redirects, then render by inspecting the raw HTML with a request such as curl -A \"Claude-User\" https://yourdomain.com, and only then read the passage on its own. Rewriting copy fixes nothing if a firewall drops the agent before it reaches the site."}}]}]}
```