Our post about Google Ads agency pricing got cited in a ChatGPT answer last month. In the same 30-day window, our logs showed hundreds of live requests for that post. The citation was visible to a reader; the requests were visible to our server. If you want to understand AI visibility, you need to look at both sides.

Here is the fact everything else follows from: an AI answer can depend on a request to a specific URL at a specific moment. It does not always happen that way. But it happens often enough that a content audit alone can miss the problem. Training crawlers and live fetchers are different bots with different jobs. They have different user agents, different robots.txt controls, and different ways to fail.

We counted those requests on our own site. One groas post drew 558 live fetches in 30 days; Claude also requested our robots.txt 1,224 times in that window. Those numbers do not tell us which request produced which answer. They do tell us where to start looking. Before you rewrite a page for AI visibility, find out whether a live fetcher can reach and read it.

First, separate the bots that learn from the bots that fetch

A training crawler gathers web content that may feed a future model. A live fetcher requests a page in connection with something happening now, such as search or a user asking for a source. One works on a broad, long horizon; the other can arrive for one URL while a user waits for an answer. Calling both of them the AI crawler hides the distinction you need to make a technical decision.

I used to talk about them as one thing. I was wrong. OpenAI documents separate bots and controls, including GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for user-initiated visits. Blocking a training bot is not the same decision as blocking a bot involved in search or a live visit. You can decline one kind of access without automatically declining the other.

That matters when someone says, “We blocked AI bots, so why did our visibility disappear?” Which bots? A rule aimed at GPTBot answers a different question from a rule aimed at OAI-SearchBot. And a user agent listed in a request is only a claim until you verify who sent it. Write rules for the job you intend to allow or refuse, not for the vague category ‘AI.’

Next, follow one question to the page request

Say someone asks ChatGPT which Google Ads agency pricing model fits a $20k/month account. To find a current source, the system may search for something like Google Ads agency pricing models percentage of spend vs flat fee 2026. Search produces candidate pages. A live fetch can then request a page the system wants to examine. That is one route from the question to an entry in your server log; it is not a guarantee that every answer triggers a fresh request.

The bot name matters here. OpenAI says ChatGPT-User is not used to determine search visibility. OAI-SearchBot has a different search-related job. If you collapse their requests into one count, you lose the very distinction that could explain why a page is reachable in one context and absent in another.

Access checks add another layer. Bots may request /robots.txt to find the rules that apply to them, and OpenAI publishes bot details and IP ranges for verification. On our site, requests for that file are steady background traffic: Claude requested it 1,224 times in 30 days. That count is not 1,224 article reads. It tells us the bot checked the rules, not what it found useful afterward. OpenAI also says to allow about 24 hours for a robots.txt change to take effect. Change a rule, then give it time before treating the next morning’s logs as a verdict.

If a fetch reaches the article, the server has to send something worth reading. A December 2024 analysis found major AI crawlers did not render JavaScript in its tests. In another example, bots saw a stripped shell of a retail page that showed size, color, price, and promotions to human visitors. Your browser’s finished page and the HTML returned to a bot can be two different documents in practice. If the answer appears only after scripts run, a successful request may still deliver too little to cite.

That is the chain: a question may lead to search; search supplies candidate URLs; access controls govern the request; the server returns a page the fetcher must be able to read. A good answer on a page does not help a live fetcher that cannot find, reach, or read the page.

Timeline from a ChatGPT question to a robots.txt check and article request

The logs show demand, not the whole answer

One groas Google Ads post received 558 live fetches in 30 days. Claude requested our robots.txt 1,224 times during that period. The 30-day crawler log teardown has the fuller breakdown. For this argument, the useful observation is simpler: some pages attract repeated live attention, and rule checks and page fetches leave different traces.

I read that as an opportunity to replace guessing with diagnosis. Logs show which URLs were requested, when, and by a verified bot if you do the verification. They do not, by themselves, show the user’s question or prove that a particular fetch produced a citation. A frequently fetched page with weak visibility is worth investigating; it is not automatically a writing failure. A page with no recorded fetches may have a discovery or access problem, but you first need to know whether your logs include requests handled at the edge.

Use the count to choose what to inspect, not to declare a cause you have not checked.

Clay model of a server rack with tiny bot figures pulling an article page

Three places the request can fail

The first break is readability. Check what the server actually sends, not just what your browser displays after the page loads. View source and look for the answer block you want a fetcher to use. If price, specifications, or the central explanation are missing from the returned HTML, put those facts in server-rendered HTML and make relevant links available there too. The JavaScript-rendering tests are a reason to test this path, not a license to assume that every bot behaves identically forever. My practical rule is less grand: do not make the answer depend on a script when you can put it in the response.

The second break is access. A permissive robots.txt file will not help if an edge rule refuses the request before it reaches the page. Cloudflare began blocking AI crawlers by default on new domains on July 1, 2025, which makes the edge configuration worth checking alongside the file. The reverse problem matters too: if you intend to refuse a user-initiated fetch, do not assume robots.txt alone enforces that refusal. Reporting on ChatGPT-User notes that it does not reliably honor those rules. Check the server or WAF behavior that actually handles the request.

Before changing either layer, make sure the request is genuine. In one analysis, 5.7% of claimed AI-crawler traffic was spoofed overall, rising to 16.7% for claimed ChatGPT-User traffic. A user agent is text supplied by the requester. Verify the IP against the bot operator’s published information before you build a plan around a spike.

The third break is discovery. A page can exist, return useful HTML, and permit a request while still being difficult for a search layer to find. Our pricing example does not begin with a bot browsing your site like a patient human. It begins with candidate URLs. An orphaned answer page, one omitted from the XML sitemap or reachable only through JavaScript filters, may never enter that candidate set. Check whether the URL appears in the sitemap, returns 200 to a plain GET, and has a link from another indexed page in ordinary HTML. These are basic checks because the failure is basic: a fetcher cannot request a candidate it was never given.

Notice the order. First ask whether the page can be found. Then ask whether the request is allowed. Then ask whether the response contains the answer. Only after those checks does a copy edit become a sensible first move.

Why a reported content gap may sit one layer lower

An AI visibility report can show that you are absent for 40 prompts while a competitor appears. The easy prescription is 40 new pages. Sometimes the report has identified a real content gap. Sometimes it has recorded the result of a failed delivery path: your existing page covers the topic, but a live fetcher cannot get a useful response from it.

Before commissioning another page, run a pre-content readability check. Look at the HTML, access rules, and edge response for the page you already have. A new article will not repair a rule that blocks the bot or a template that sends only a shell. If those layers work and the page still fails to answer the question clearly, then rewrite it. Diagnose the layer before you pay to improve the wrong one.

Run a 30-day log test before you touch the copy

Here is the question for the test: when someone seeks an answer about what you sell, do verified AI bots request the pages that answer it? We cannot recover every user question from a server log. We can test the request path instead.

Pull 30 days of edge logs, not only origin logs, because origin logs can miss requests served from CDN cache. Keep both sets if you have them. Choose the URLs you would expect to answer your important questions, including the pricing page in our running example. Hold that list steady while you inspect the requests; otherwise it is too easy to declare whichever page received traffic the one you meant to test.

I expect to find many robots.txt requests, fewer verified page fetches, and attention concentrated on a handful of URLs. That is an expectation, not a result I can promise for your site. The full 30-day server log test goes further. For this pass, work in this order:

  1. Find candidates. Filter for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, and PerplexityBot. Treat that user-agent search as a starting point, not a count you can publish yet.
  2. Separate checks from reads. Put /robots.txt requests in one bucket and article requests in another. Record the requested URL and the response status for each page hit. A rule check is not evidence that the bot read your answer.
  3. Verify the requester. Compare the claimed identity with published IP ranges or use reverse and forward DNS verification where appropriate. Check the string and the network identity: spoofed user agents can inflate the apparent demand.
  4. Sort by job and URL. Keep training crawlers separate from search-related and user-initiated fetchers. Then compare verified live requests with the specific pages you chose at the start.

Now interpret the result without asking the logs to say more than they can. If a target page has verified live fetches, inspect the returned HTML and the answer itself before deciding what to change. If you see rule checks but no successful page responses, examine access and edge behavior. If you see no relevant requests in complete edge logs, start with discovery, rules, and whether the page is a plausible candidate for the question. Citations without a fetch in your window do not prove that last week’s edit worked; the answer may not have required a fresh request in that window.

Either outcome changes the work order. A broken request path calls for a technical fix. A readable, reachable page that does not answer the question calls for better copy. Writing first feels productive, but it is a poor test of a delivery problem.

Put the model to work on one more case

Say you run a home services company and ChatGPT keeps recommending a competitor. Your first instinct is to write a better services page. I would pause and follow the chain we just built.

Can a search layer discover your page through the sitemap and plain HTML links? Does a plain GET return the service area and the answer you want a reader to see, rather than a shell that fills in later? Do robots.txt and the edge configuration permit the bot whose access matters to you? Finally, do complete edge logs show verified live requests for that URL during the period you are examining?

If the chain breaks before the page can be read, rewriting the headline does not mend it. Fix discovery, access, or rendering at the point that failed, then look again. If the request succeeds and the returned page still dodges the buyer’s question, write the better answer. That is not a shortcut around content work. It is how you avoid doing content work in place of the technical fix you actually need.

I check logs before I check copy for that reason. Training crawlers and live fetchers do different jobs, and only the latter can show you part of the path a fresh answer might take through your site. Count the right requests, verify them, and find the broken link in the chain. Then write.