--- title: "13,405 Requests, Zero Live-Session Fetches: The Guide AI Agents Kept Requesting" description: "Our 30-day edge logs show thousands of requests for site controls and category hubs, but 273 verified live-session fetches for one YouTube ads guide. Here is what that contrast can—and cannot—tell us." url: "https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2ef2d9d5-92d3-4af8-b2a7-e82eec4bf7f2.png" published: "2026-10-11T05:35:44.450Z" modified: "2026-10-11T05:35:44.524Z" --- October 11, 2026 · 9 min read # 13,405 Requests, Zero Live-Session Fetches: The Guide AI Agents Kept Requesting [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![A model building where a huge crowd mobs the front gate reading a notice, while a small queue walks past an empty hall to one lit room handing out a booklet.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2ef2d9d5-92d3-4af8-b2a7-e82eec4bf7f2.png) In this article 1. [The log had two very different kinds of traffic](#the-log-had-two-very-different-kinds-of-traffic) 2. [The bots reached the hubs, then went elsewhere](#the-bots-reached-the-hubs-then-went-elsewhere) 3. [Three folders down, the requests kept coming](#three-folders-down-the-requests-kept-coming) 4. [I went back to the permissions file](#i-went-back-to-the-permissions-file) At the top of our Cloudflare edge logs sat `/robots.txt`: **13,405 requests in 30 days**, and not one verified live-session fetch. Three folders down, a YouTube ads guide drew 273 verified fetches from agents serving live ChatGPT and Perplexity sessions. The first number looked impressive until I put it beside the second. Across 720 hours, automated user agents kept asking for our thirty-line permissions file. `Claude-User` alone requested it 1,160 times. An unfiltered dashboard could turn that into a handsome chart of “AI engagement.” I used to manage accounts where a chart like that could survive an entire client meeting. It would still be the wrong chart. ## The log had two very different kinds of traffic Before following any bot through the site, I separated raw requests from requests whose IP addresses matched published vendor ranges. I also separated background crawlers from agents that fetch pages during live user sessions. Those distinctions changed the story. These figures come from our own Cloudflare edge logs over the same thirty-day window. **A verified fetch is not a visible citation**: the logs show a page request, not what appeared in the finished chat answer. | Page | Total log requests | Verified AI fetches | Live-session fetches | | :-------------------------------------------------------------------------------------- | -----------------: | ------------------: | -------------------: | | `/robots.txt` | 13,405 | 2,190 | 0 | | `/services/google-ads` | 3,512 | 274 | 0 | | `/seo-ai-search` | 3,480 | 268 | 0 | | [YouTube ads frequency guide](https://groas.com/post/youtube-ads-frequency-policy-2026) | 482 | 294 | 273 | | Google Ads updates changelog | 310 | 182 | 139 | | Technical campaign setup guide | 245 | 140 | 112 | That last column is the one I kept returning to. It does not tell us whether a reader saw a footnote, clicked it, or bought anything. It tells us which of these pages the verified user-triggered agents requested while handling live sessions. The two category hubs had hundreds of verified AI fetches between them and **zero in that column**. The YouTube guide had 273. The columns are not interchangeable measures of popularity. Total requests include traffic whose claimed identity I could not verify. Verified AI fetches include background activity. Even the live-session column counts requests, not people or citations. Keeping all three columns in view makes the contrast less flattering to the busiest URLs, but considerably more useful when deciding which pages deserve a closer look. There was another reason not to trust the first chart I saw: **74% of requests claiming to be AI bots in our logs were unverified**. A user-agent label is text in an HTTP header; it is not identification. [Search Engine Land’s guide to AI crawlers](https://searchengineland.com/guide/ai-crawlers) describes the spoofing problem. If I had reported every claimed `ChatGPT-User` request as a real one, I would have mixed vendor traffic with whoever else chose to use that name. The distinction between fleets matters, too. [OpenAI’s bot documentation](https://developers.openai.com/api/docs/bots) distinguishes `GPTBot`, which crawls for model training, from `ChatGPT-User`, which fetches pages in response to user requests. [Perplexity documents](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) a similar split between `PerplexityBot` and `Perplexity-User`. [Anthropic’s crawler documentation](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) explains the role of `Claude-User`. A request for `/robots.txt` can be part of checking what a bot is allowed to access. It is not evidence that anyone received an answer from our site. Outside our logs, [Cloudflare Radar reported](https://blog.cloudflare.com/control-content-use-for-ai-training/) roughly 1,700 OpenAI crawls per referred visitor and up to 73,000 for Anthropic, against 10 to 14 for Google. Those are Cloudflare’s broader figures, not a conversion rate for groas. They illustrate why a pile of requests should not be mistaken for a pile of readers. For our pages, the narrower live-session column was the useful place to look. ## The bots reached the hubs, then went elsewhere After the permissions file, the apparent winners were familiar: broad category pages. One hub collected 3,512 total requests; another collected 3,480. Roughly 270 requests to each matched official AI vendor IP ranges. If I had stopped at verified crawler activity, I could have said the hubs were getting attention and been technically right. But across those 720 hours, neither hub recorded a verified `ChatGPT-User` or `Perplexity-User` fetch. The bots came through. When the user-triggered agents fetched pages from this group, they went elsewhere. ![Website hierarchy fading into fog beside one illuminated document.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/578c7d4a-2939-41f2-b172-eb605e3a6584.png) I understand why we built the hubs. A person browsing a site needs routes into a topic, and a category page can organize them. The problem starts when we treat that navigational job as proof that the same page will answer a precise question. A hub introduces several subjects and points outward. It may be useful precisely because it does _not_ settle every one of them on the spot. **The prompt is narrow; the hub is a map.** That is a poor match for someone asking an assistant about a particular advertising constraint. We cannot see the private ranking decisions behind every request in these logs, so I will not claim to know why either hub was passed over. I can see that the verified live-session agents did not fetch them during this window, while they repeatedly fetched narrower pages. That does not make the hubs failed pages. It means their request totals answer a different question from the one I had opened the logs to ask. If I want to know whether a page helps someone move around the site, a hub belongs in that discussion. If I want to know what user-triggered agents retrieved, I have to follow those agents past the hub. Otherwise I am grading the map as though it were the destination. That pattern is not unique to our little slice of the web. Research on live AI search results found that [82.5% of cited sources were deeply nested, narrow informational pages](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG_GDXNQ3gGeHr0ygmXeekEvK833etQPKsukSXZoOFI_b4FFJNLZSfIivflfqGrrlvswkP3bNW49nNJgxPHCt1mhObSFr_bM5LhPUv0ScyaltZ42bR9h5bODTN1W1oqPRe0W31Bp6k9l4kRiYh4TXyuU4XmBamungbjDnw-7kbjtcMT51SK9w=). That figure describes the cited sources in that research, not the citation rate of our hubs. It did, however, give me a reason to keep following the requests past the category level. ## Three folders down, the requests kept coming The standout was our [_YouTube Ads in 2026_ guide](https://groas.com/post/youtube-ads-frequency-policy-2026), a post about frequency changes, policy updates, and formats. It was not our longest article, and we had not supported it with paid distribution. In the same thirty-day window, its URL received 482 total log requests. We verified 294 as AI-vendor fetches; **273 came from live-session agents**: 256 from `ChatGPT-User` and 17 from `Perplexity-User`. I cannot read the users’ prompts from an edge log. I cannot say that each fetch produced a citation, much less that 273 different people read one. What I can say is that the agents repeatedly requested this specific document while handling live sessions. They did not do that for the broad category hubs in the table. Two other pages followed the same path. Our running Google Ads updates changelog recorded 139 verified live-session fetches, and a technical campaign setup guide recorded 112. Neither was a general tour of everything we know about search marketing. Each dealt with a bounded operational subject. Once I looked past total traffic, those three pages—not the hubs—were where the activity gathered. The comparison matters because all of these URLs sit in the same thirty-day view. I am not setting a recent guide against an old snapshot of a hub, or treating all bot traffic as though it came from the same kind of agent. That still does not establish _why_ one page was requested more often. It does let me put the pages next to each other and ask what a reader looking for an answer would find. The difference became clearer when I read the pages as someone in a hurry would. The YouTube post gets to its frequency and format questions early. The changelog is tied to dated updates. The setup guide deals in operational steps rather than a long preface about why campaigns matter. They give a reader somewhere specific to land. I have written the other kind of opening. First you define the industry, then announce that it is evolving, then spend several paragraphs warming up before you answer the question in the headline. It feels comprehensive while you are writing it. To the person who needs the rule, it feels like waiting on hold. ![Cutaway diagram contrasting a dated answer block with a broad pillar page.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/19d84d9e-a197-4b90-8f5e-e8712b6af598.png) The three frequently fetched pages shared a few useful traits: - **The answer arrived near the top.** A reader looking for the relevant rule or change did not have to pass through a broad introduction first. - **The details were concrete and dated.** Caps, rollout timing, and bidding constraints did more work than adjectives about what was “essential.” - **The subject stayed bounded.** The YouTube guide did not also try to teach video editing or agency selection. It dealt with the advertising question it had set out to answer. Those observations do not prove which sentence an assistant used. They do tell me what sort of page it kept requesting. Research on [Generative Engine Optimization](https://peec.ai/ai-search-geo-statistics) has also reported citation-visibility gains from concrete statistics and authoritative citations in a 10,000-query benchmark. That is a different test, with different limits, not an explanation of our 273 fetches. I mention it because its practical direction matches what was on the pages: specific information a reader can use, rather than another synonym for “effective.” ## I went back to the permissions file By this point, `/robots.txt` looked different. Its 13,405 requests no longer suggested that the site was winning anyone’s attention. The file was doing its job as a permissions file. The category hubs were doing their job as navigation. Neither job was the one performed by the narrow guides that appeared in the live-session column. **I would start in the edge logs**, not with a chart of referral traffic or a screenshot of a chatbot answer. Separate the background agents from `ChatGPT-User` and `Perplexity-User`, then check claimed identities against the vendors’ published IP ranges. Look at the URLs left after that filter. [Search Engine Land’s discussion of user-triggered edge requests](https://searchengineland.com/proxies-for-prompts-466351) is useful context for what this method measures, and for what it does not. I would keep the broad pages in the comparison rather than quietly dropping them. Seeing a hub with verified crawler requests but no verified live-session fetches told me something different from seeing a hub with no verified requests at all. One invites a closer look at what the page answers; the other invites a check of whether agents can get to it. The log cannot make either decision for me, but it can stop me from solving the wrong problem. If those agents never fetch a page you expected them to use, check whether they can reach and read it before commissioning more copy. Our [45-minute site-readability check](https://groas.com/post/can-chatgpt-even-read-your-site-a-45-min) walks through that diagnostic. A page blocked from retrieval cannot contribute its answer. A page that _is_ fetched still has to earn its place in the final response; the access log cannot settle that part. Then I would look at the questions buyers actually bring to you: the awkward ones in search terms, sales qualification transcripts, and support logs. Pick a question with a concrete answer. Put the answer, its date where the date matters, and its constraints near the top. Do not make someone wade through a 4,000-word tour of neighboring topics to find the paragraph they came for. That is the decision these numbers support for our next page, not a promise that every narrow post will be cited. An hour before I finished this, I checked the logs again. Another 418 requests had hit `/robots.txt` in the preceding twenty-four hours. On that same day, `ChatGPT-User` had fetched the dated YouTube guide 31 times. The permissions checks kept coming. Three folders down, so did the requests for the answer. ## Frequently Asked Questions ### Does a high number of /robots.txt requests mean AI agents are engaging with my site content? No. In the described 30-day Cloudflare logs, /robots.txt received 13,405 requests and not one verified live-session fetch. A request for the permissions file only reflects a bot checking what it is allowed to access; it is not evidence that anyone received an answer drawn from the site. ### Can I trust the user-agent label in my logs to tell me which bot visited? No. A user-agent label is text in an HTTP header, not identification, and 74% of requests claiming to be AI bots in these logs were unverified. Claims should be checked against the vendors' published IP ranges to separate real vendor traffic from spoofed requests. ### What is the difference between GPTBot and ChatGPT-User? GPTBot crawls the web for model training, while ChatGPT-User fetches pages in response to live user requests. Perplexity and Anthropic have similar splits between PerplexityBot and Perplexity-User, and Claude-User. Only the user-triggered agents indicate that a page was fetched while serving an actual user session. ### Why did the broad category hub pages get no live-session fetches from AI agents? In the 30-day window, the two category hubs collected hundreds of verified AI fetches and zero verified ChatGPT-User or Perplexity-User fetches. A hub introduces several subjects and points outward, which suits navigation, while user-triggered agents looked for narrow pages that answer a specific question. The article does not claim to know the private ranking reasons behind this. ### What kind of pages did live-session AI agents actually request most often? Narrow, dated pages that answer specific questions: the YouTube ads frequency guide got 273 verified live-session fetches (256 from ChatGPT-User and 17 from Perplexity-User), a Google Ads updates changelog got 139, and a technical campaign setup guide got 112. These pages put the answer near the top, used concrete dated details, and stayed on a bounded subject. ### How should I measure whether AI agents are using my pages? Start in the edge logs rather than referral traffic charts or chatbot screenshots. Separate background crawlers from ChatGPT-User and Perplexity-User, verify claimed identities against vendors' published IP ranges, and look at the URLs remaining after that filter. Then pick a real buyer question and put its concrete answer, with dates where relevant, near the top of the page. ### What should I do if AI agents never fetch a page I expected them to use? Check whether the agents can reach and read the page before commissioning more copy, since a blocked page cannot contribute its answer. Note that a fetched page still has to earn its place in the final response; the access log cannot settle that part. ## Pay For Results, Not For Hours Businesses buy the outcome, agencies resell it, and groas answers for it either way. [See If You Qualify](https://groas.typeform.com/to/xC1bQNUT) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#webpage","url":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30","name":"13,405 Requests, Zero Live-Session Fetches: The Guide AI Agents Kept Requesting","description":"Our 30-day edge logs show thousands of requests for site controls and category hubs, but 273 verified live-session fetches for one YouTube ads guide. Here is what that contrast can—and cannot—tell us.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2ef2d9d5-92d3-4af8-b2a7-e82eec4bf7f2.png"},"breadcrumb":{"@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#breadcrumb"},"dateModified":"2026-10-11T05:35:44.524Z","mainEntity":{"@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"13,405 Requests, Zero Live-Session Fetches: The Guide AI Agents Kept Requesting","item":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#article","url":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30","mainEntityOfPage":{"@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"13,405 Requests, Zero Live-Session Fetches: The Guide AI Agents Kept Requesting","description":"Our 30-day edge logs show thousands of requests for site controls and category hubs, but 273 verified live-session fetches for one YouTube ads guide. Here is what that contrast can—and cannot—tell us.","datePublished":"2026-10-11T05:35:44.450Z","dateModified":"2026-10-11T05:35:44.524Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2ef2d9d5-92d3-4af8-b2a7-e82eec4bf7f2.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"requests, fetches, guide, agents, live-session, verified, pages, live-session fetches","wordCount":1906,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/the-page-chatgpt-kept-coming-back-to-30#faq","mainEntity":[{"@type":"Question","name":"Does a high number of /robots.txt requests mean AI agents are engaging with my site content?","acceptedAnswer":{"@type":"Answer","text":"No. In the described 30-day Cloudflare logs, /robots.txt received 13,405 requests and not one verified live-session fetch. A request for the permissions file only reflects a bot checking what it is allowed to access; it is not evidence that anyone received an answer drawn from the site."}},{"@type":"Question","name":"Can I trust the user-agent label in my logs to tell me which bot visited?","acceptedAnswer":{"@type":"Answer","text":"No. A user-agent label is text in an HTTP header, not identification, and 74% of requests claiming to be AI bots in these logs were unverified. Claims should be checked against the vendors' published IP ranges to separate real vendor traffic from spoofed requests."}},{"@type":"Question","name":"What is the difference between GPTBot and ChatGPT-User?","acceptedAnswer":{"@type":"Answer","text":"GPTBot crawls the web for model training, while ChatGPT-User fetches pages in response to live user requests. Perplexity and Anthropic have similar splits between PerplexityBot and Perplexity-User, and Claude-User. Only the user-triggered agents indicate that a page was fetched while serving an actual user session."}},{"@type":"Question","name":"Why did the broad category hub pages get no live-session fetches from AI agents?","acceptedAnswer":{"@type":"Answer","text":"In the 30-day window, the two category hubs collected hundreds of verified AI fetches and zero verified ChatGPT-User or Perplexity-User fetches. A hub introduces several subjects and points outward, which suits navigation, while user-triggered agents looked for narrow pages that answer a specific question. The article does not claim to know the private ranking reasons behind this."}},{"@type":"Question","name":"What kind of pages did live-session AI agents actually request most often?","acceptedAnswer":{"@type":"Answer","text":"Narrow, dated pages that answer specific questions: the YouTube ads frequency guide got 273 verified live-session fetches (256 from ChatGPT-User and 17 from Perplexity-User), a Google Ads updates changelog got 139, and a technical campaign setup guide got 112. These pages put the answer near the top, used concrete dated details, and stayed on a bounded subject."}},{"@type":"Question","name":"How should I measure whether AI agents are using my pages?","acceptedAnswer":{"@type":"Answer","text":"Start in the edge logs rather than referral traffic charts or chatbot screenshots. Separate background crawlers from ChatGPT-User and Perplexity-User, verify claimed identities against vendors' published IP ranges, and look at the URLs remaining after that filter. Then pick a real buyer question and put its concrete answer, with dates where relevant, near the top of the page."}},{"@type":"Question","name":"What should I do if AI agents never fetch a page I expected them to use?","acceptedAnswer":{"@type":"Answer","text":"Check whether the agents can reach and read the page before commissioning more copy, since a blocked page cannot contribute its answer. Note that a fetched page still has to earn its place in the final response; the access log cannot settle that part."}}]}]} ```