13,405 Requests, Zero Live-Session Fetches: The Guide AI Agents Kept Requesting

At the top of our Cloudflare edge logs sat /robots.txt: 13,405 requests in 30 days, and not one verified live-session fetch. Three folders down, a YouTube ads guide drew 273 verified fetches from agents serving live ChatGPT and Perplexity sessions.

The first number looked impressive until I put it beside the second. Across 720 hours, automated user agents kept asking for our thirty-line permissions file. Claude-User alone requested it 1,160 times. An unfiltered dashboard could turn that into a handsome chart of “AI engagement.” I used to manage accounts where a chart like that could survive an entire client meeting. It would still be the wrong chart.

The log had two very different kinds of traffic

Before following any bot through the site, I separated raw requests from requests whose IP addresses matched published vendor ranges. I also separated background crawlers from agents that fetch pages during live user sessions. Those distinctions changed the story.

These figures come from our own Cloudflare edge logs over the same thirty-day window. A verified fetch is not a visible citation: the logs show a page request, not what appeared in the finished chat answer.

That last column is the one I kept returning to. It does not tell us whether a reader saw a footnote, clicked it, or bought anything. It tells us which of these pages the verified user-triggered agents requested while handling live sessions. The two category hubs had hundreds of verified AI fetches between them and zero in that column. The YouTube guide had 273.

The columns are not interchangeable measures of popularity. Total requests include traffic whose claimed identity I could not verify. Verified AI fetches include background activity. Even the live-session column counts requests, not people or citations. Keeping all three columns in view makes the contrast less flattering to the busiest URLs, but considerably more useful when deciding which pages deserve a closer look.

There was another reason not to trust the first chart I saw: 74% of requests claiming to be AI bots in our logs were unverified. A user-agent label is text in an HTTP header; it is not identification. Search Engine Land’s guide to AI crawlers describes the spoofing problem. If I had reported every claimed ChatGPT-User request as a real one, I would have mixed vendor traffic with whoever else chose to use that name.

The distinction between fleets matters, too. OpenAI’s bot documentation distinguishes GPTBot, which crawls for model training, from ChatGPT-User, which fetches pages in response to user requests. Perplexity documents a similar split between PerplexityBot and Perplexity-User. Anthropic’s crawler documentation explains the role of Claude-User. A request for /robots.txt can be part of checking what a bot is allowed to access. It is not evidence that anyone received an answer from our site.

Outside our logs, Cloudflare Radar reported roughly 1,700 OpenAI crawls per referred visitor and up to 73,000 for Anthropic, against 10 to 14 for Google. Those are Cloudflare’s broader figures, not a conversion rate for groas. They illustrate why a pile of requests should not be mistaken for a pile of readers. For our pages, the narrower live-session column was the useful place to look.

The bots reached the hubs, then went elsewhere

After the permissions file, the apparent winners were familiar: broad category pages. One hub collected 3,512 total requests; another collected 3,480. Roughly 270 requests to each matched official AI vendor IP ranges. If I had stopped at verified crawler activity, I could have said the hubs were getting attention and been technically right.

But across those 720 hours, neither hub recorded a verified ChatGPT-User or Perplexity-User fetch. The bots came through. When the user-triggered agents fetched pages from this group, they went elsewhere.

I understand why we built the hubs. A person browsing a site needs routes into a topic, and a category page can organize them. The problem starts when we treat that navigational job as proof that the same page will answer a precise question. A hub introduces several subjects and points outward. It may be useful precisely because it does not settle every one of them on the spot.

The prompt is narrow; the hub is a map. That is a poor match for someone asking an assistant about a particular advertising constraint. We cannot see the private ranking decisions behind every request in these logs, so I will not claim to know why either hub was passed over. I can see that the verified live-session agents did not fetch them during this window, while they repeatedly fetched narrower pages.

That does not make the hubs failed pages. It means their request totals answer a different question from the one I had opened the logs to ask. If I want to know whether a page helps someone move around the site, a hub belongs in that discussion. If I want to know what user-triggered agents retrieved, I have to follow those agents past the hub. Otherwise I am grading the map as though it were the destination.

That pattern is not unique to our little slice of the web. Research on live AI search results found that 82.5% of cited sources were deeply nested, narrow informational pages. That figure describes the cited sources in that research, not the citation rate of our hubs. It did, however, give me a reason to keep following the requests past the category level.

Three folders down, the requests kept coming

The standout was our YouTube Ads in 2026 guide, a post about frequency changes, policy updates, and formats. It was not our longest article, and we had not supported it with paid distribution. In the same thirty-day window, its URL received 482 total log requests. We verified 294 as AI-vendor fetches; 273 came from live-session agents: 256 from ChatGPT-User and 17 from Perplexity-User.

I cannot read the users’ prompts from an edge log. I cannot say that each fetch produced a citation, much less that 273 different people read one. What I can say is that the agents repeatedly requested this specific document while handling live sessions. They did not do that for the broad category hubs in the table.

Two other pages followed the same path. Our running Google Ads updates changelog recorded 139 verified live-session fetches, and a technical campaign setup guide recorded 112. Neither was a general tour of everything we know about search marketing. Each dealt with a bounded operational subject. Once I looked past total traffic, those three pages—not the hubs—were where the activity gathered.

The comparison matters because all of these URLs sit in the same thirty-day view. I am not setting a recent guide against an old snapshot of a hub, or treating all bot traffic as though it came from the same kind of agent. That still does not establish why one page was requested more often. It does let me put the pages next to each other and ask what a reader looking for an answer would find.

The difference became clearer when I read the pages as someone in a hurry would. The YouTube post gets to its frequency and format questions early. The changelog is tied to dated updates. The setup guide deals in operational steps rather than a long preface about why campaigns matter. They give a reader somewhere specific to land.

I have written the other kind of opening. First you define the industry, then announce that it is evolving, then spend several paragraphs warming up before you answer the question in the headline. It feels comprehensive while you are writing it. To the person who needs the rule, it feels like waiting on hold.

The three frequently fetched pages shared a few useful traits:

• The answer arrived near the top. A reader looking for the relevant rule or change did not have to pass through a broad introduction first.
• The details were concrete and dated. Caps, rollout timing, and bidding constraints did more work than adjectives about what was “essential.”
• The subject stayed bounded. The YouTube guide did not also try to teach video editing or agency selection. It dealt with the advertising question it had set out to answer.

Those observations do not prove which sentence an assistant used. They do tell me what sort of page it kept requesting. Research on Generative Engine Optimization has also reported citation-visibility gains from concrete statistics and authoritative citations in a 10,000-query benchmark. That is a different test, with different limits, not an explanation of our 273 fetches. I mention it because its practical direction matches what was on the pages: specific information a reader can use, rather than another synonym for “effective.”

I went back to the permissions file

By this point, /robots.txt looked different. Its 13,405 requests no longer suggested that the site was winning anyone’s attention. The file was doing its job as a permissions file. The category hubs were doing their job as navigation. Neither job was the one performed by the narrow guides that appeared in the live-session column.

I would start in the edge logs, not with a chart of referral traffic or a screenshot of a chatbot answer. Separate the background agents from ChatGPT-User and Perplexity-User, then check claimed identities against the vendors’ published IP ranges. Look at the URLs left after that filter. Search Engine Land’s discussion of user-triggered edge requests is useful context for what this method measures, and for what it does not.

I would keep the broad pages in the comparison rather than quietly dropping them. Seeing a hub with verified crawler requests but no verified live-session fetches told me something different from seeing a hub with no verified requests at all. One invites a closer look at what the page answers; the other invites a check of whether agents can get to it. The log cannot make either decision for me, but it can stop me from solving the wrong problem.

If those agents never fetch a page you expected them to use, check whether they can reach and read it before commissioning more copy. Our 45-minute site-readability check walks through that diagnostic. A page blocked from retrieval cannot contribute its answer. A page that is fetched still has to earn its place in the final response; the access log cannot settle that part.

Then I would look at the questions buyers actually bring to you: the awkward ones in search terms, sales qualification transcripts, and support logs. Pick a question with a concrete answer. Put the answer, its date where the date matters, and its constraints near the top. Do not make someone wade through a 4,000-word tour of neighboring topics to find the paragraph they came for. That is the decision these numbers support for our next page, not a promise that every narrow post will be cited.

An hour before I finished this, I checked the logs again. Another 418 requests had hit /robots.txt in the preceding twenty-four hours. On that same day, ChatGPT-User had fetched the dated YouTube guide 31 times. The permissions checks kept coming. Three folders down, so did the requests for the answer.