---
title: "30 Days of AI Bot Logs: Thousands of Hits, Zero Answers for Category Pages"
description: "Our homepage logged 11,441 hits claiming to be AI bots in 30 days. Only 2,187 were verified."
url: "https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our"
image: "https://groas.com/media/blog/82ca40907f165aa6ef6baa0903a5a70019e13bb782846dbcacfd92d25485cc72.png"
published: "2026-10-06T05:33:22.844Z"
modified: "2026-10-06T05:33:23.349Z"
---

[AI For Google Ads](https://groas.com/category/ai-for-google-ads) · October 6, 2026 · 10 min read

# 30 Days of AI Bot Logs: Thousands of Hits, Zero Answers for Category Pages

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)

![Model town: a crowd mobs a giant signpost and leaves empty-handed, while a small red booth with a calendar hands out paper answers to a short queue.](https://groas.com/media/blog/82ca40907f165aa6ef6baa0903a5a70019e13bb782846dbcacfd92d25485cc72.png)

In this article

1. [What the edge logs count, and what they cannot see](#what-the-edge-logs-count-and-what-they-cannot-see)
2. [Finding 1: the homepage’s 11,441 claimed hits became 2,187 verified hits](#finding-1-the-homepages-11441-claimed-hits-became-2187-verified-hits)
3. [Finding 2: category pages got thousands of hits and fed zero answers](#finding-2-category-pages-got-thousands-of-hits-and-fed-zero-answers)
4. [Finding 3: one dated guide fed 273 live answers](#finding-3-one-dated-guide-fed-273-live-answers)
5. [Finding 4: permission checks are traffic, not citations](#finding-4-permission-checks-are-traffic-not-citations)
6. [The decision: fix the measurement, then fix one page](#the-decision-fix-the-measurement-then-fix-one-page)
7. [The limits: one site, top paths, 30 days](#the-limits-one-site-top-paths-30-days)

Our homepage logged **11,441 hits claiming to be AI bots in 30 days. Only 2,187 were verified.** I used to tell clients to watch bot hits in the logs as a proxy for AI visibility. I was wrong.

The rest of the month made the distinction harder to ignore. Category pages drew more than 3,000 claimed hits each and fed **zero live answers**. One specific, dated YouTube ads guide had 294 verified fetches; 273 fed live ChatGPT and Claude answers. AI attention and AI citation are different things. Our logs point to a page-shape problem, not a crawl-volume problem.

## What the edge logs count, and what they cannot see

This is a 30-day slice of groas.com edge logs, not Google Search Console and not a rank tracker. I split the activity into three groups:

1. **Claimed AI hits:** requests carrying an AI bot user-agent.
2. **Verified hits:** requests whose IP matched the relevant vendor’s published list.
3. **Live-answer fetches:** user-triggered retrieval requests associated with a live ChatGPT, Claude, or Perplexity answer.

Those are different denominators. [Any script can claim to be ChatGPT-User in its user-agent](https://www.honeyb.ai/blog/gptbot-crawler-guide). IP verification gets us closer to genuine vendor traffic. It does not make every verified request an answer, and an answer-related fetch does not, by itself, prove that a reader saw a visible citation. The logs tell us which pages fed live answers. They cannot show the full text of every answer or settle why a model chose one page over another.

The distinction matters before we even compare pages. Training crawlers gather material for models; retrieval agents fetch material in response to a question. [Counting training and retrieval bots together](https://github.com/ravigupta0210/seo-auditor/blob/HEAD/docs/launch/devto/what-is-an-ai-crawler.md) turns two different jobs into one impressive-looking traffic line. Impressive-looking is not a reporting category I trust.

Verification also needs to be maintained, not performed once and forgotten. A separate two-week sample across 16 AI crawlers found that [5.7% of traffic claiming an AI crawler identity was spoofed](https://www.humansecurity.com/learn/blog/ai-crawler-spoofing-chatgpt-mistral-perplexity/); about one in six requests claiming to be ChatGPT-User was fake. That is not the spoofing rate in our logs. It is a useful reason to check identity rather than trust a label. For OpenAI, that means [matching IPs against its per-agent lists](https://www.honeyb.ai/blog/gptbot-crawler-guide) on a refresh schedule. For Anthropic, [the published list has changed from its older tokens](https://nohacks.co/blog/ai-user-agents-landscape-2026). A report built on user-agents alone counts strangers knocking on the door as invited guests.

## Finding 1: the homepage’s 11,441 claimed hits became 2,187 verified hits

Here is the cut that changed how I read a bot report. Of 11,441 homepage hits claiming an AI identity, 2,187 survived IP verification: **19.1%**. Category pages had a lower verified share, roughly **7–8%**. Their 3,000-plus figures are _claimed hits_, not verified fetches. That distinction is easy to lose when both appear on a chart labelled “AI traffic.”

| Page group     | Claimed AI hits | Verified hits           | Verified share |
| -------------- | --------------- | ----------------------- | -------------- |
| Homepage       | 11,441          | 2,187                   | 19.1%          |
| Category pages | 3,000+ each     | Not reported as a count | \~7–8%         |

Every figure in this table comes from the same 30-day groas edge-log slice. The category-page range is approximate; the dataset here does not give a per-page verified count to put in that column.

Say you spend $20k a month and get a report celebrating a climb in crawler hits. Before anyone calls it visibility, ask whether those hits were checked against vendor IPs. On our homepage, roughly four out of five claimed hits did not pass that check. I would not apply that exact fraction to every site, or even to every page on ours. I would stop treating an unchecked user-agent chart as evidence that an AI service is paying attention.

**First verify the visitor. Then decide whether the visit means anything.**

## Finding 2: category pages got thousands of hits and fed zero answers

The category pages each drew more than 3,000 claimed bot hits during the window. **None fed a live answer in this dataset.** Same site, same month, very different outcome from the dated guides. A hit says something requested a URL. It does not tell you whether the request came from a genuine AI agent, whether that agent was gathering training material, or whether a user had asked a question.

The bot’s job matters here. OpenAI uses separate `robots.txt` tokens for GPTBot, OAI-SearchBot, and ChatGPT-User: training, search indexing, and a page pull related to a user request. [Blocking GPTBot is not the same as blocking OAI-SearchBot](https://www.honeyb.ai/blog/gptbot-crawler-guide). Perplexity likewise distinguishes [PerplexityBot, which indexes, from Perplexity-User, which fetches for a user](https://docs.perplexity.ai/guides/bots). One total labelled “AI crawls” hides decisions that affect different parts of the path to an answer.

| Bot type                                   | Job                      | Relationship to an answer                               |
| ------------------------------------------ | ------------------------ | ------------------------------------------------------- |
| GPTBot, ClaudeBot                          | Training crawl           | Not a live-answer fetch                                 |
| OAI-SearchBot, PerplexityBot               | Search indexing          | Helps make pages retrievable                            |
| ChatGPT-User, Claude-User, Perplexity-User | User-triggered retrieval | Can feed a live answer; not proof of a visible citation |

The table describes bot roles, not counts from our logs. It also explains why I would not respond to the category-page result by chasing more raw crawls. More requests of the wrong kind would make the top line bigger while leaving the answer line at zero.

**Report training, indexing, and live retrieval separately.** Otherwise, the number most likely to impress someone in a meeting is the least useful one for this decision.

## Finding 3: one dated guide fed 273 live answers

The YouTube ads 2026 guide had **294 verified fetches, 273 of which fed a live answer**. That is 92.9% of its verified fetches in this 30-day slice. The Google Ads updates guide fed 106 live answers, and the AI Max guide fed 139. We do not have verified-fetch totals for those last two guides in this cut, so I cannot give them comparable rates.

| Page                     | Verified fetches | Live-answer fetches | Share of verified fetches feeding answers |
| ------------------------ | ---------------- | ------------------- | ----------------------------------------- |
| YouTube ads 2026 guide   | 294              | 273                 | 92.9%                                     |
| AI Max guide             | Not available    | 139                 | Not available                             |
| Google Ads updates guide | Not available    | 106                 | Not available                             |
| Category pages           | Not available    | 0                   | Not available                             |

These are groas edge-log counts. The 3,000-plus category-page hits in the previous table were _claimed_ hits, so dividing zero answers by them would not produce the same rate shown for the YouTube guide. Zero is still zero. The missing denominators are still missing.

What did the pages feeding answers have that the category pages did not? The guides name a specific subject, carry a visible date, and put numbers, steps, and definitions in the body. A category page lists routes to other pages. If a retrieval agent needs material for a narrow question, the guide offers an answer it can use; the index asks it to keep looking. That is a plausible explanation for the gap, not a controlled test proving that a year in a title caused 273 answers.

Freshness deserves attention, with the same caution. One analysis of generative-AI citations reports that [95% of cited pages had been updated within 10 months and pages with a visible last-updated date earned 1.8 times more citations](https://www.airops.com/blog/how-often-refresh-content). Another synthesis reports [79% day-to-day churn in ChatGPT sources and says more than 70% of cited pages had been updated within the previous 12 months](https://www.linkedin.com/pulse/heres-what-ai-actually-looks-before-citing-your-content-rohit-rajpal-qaxbf). Those are outside analyses, not measurements from our edge logs, and neither turns a date stamp into a guaranteed citation.

I used to treat the date stamp as cosmetic. Now I would check whether the date reflects a real update, then refresh the numbers and steps that changed. Rewriting an introduction while leaving stale facts in place is tidying the shop window and ignoring the shelves. **Make the answer specific and current; do not merely make the page look current.**

## Finding 4: permission checks are traffic, not citations

Our `robots.txt` received **1,160 Claude-User requests classified as live-answer fetches** in the window. Nobody needs a quote from a `robots.txt` file in a ChatGPT answer. That count is a reminder that a retrieval path can generate requests to supporting files as well as to pages a reader might actually see.

I would not count those 1,160 requests as 1,160 citations, or use them to calculate a page’s answer rate. I would use them to check a less glamorous part of the setup: is the file available, current, and explicit about the agents we intend to allow? A stale block can interfere with the path we want open. Guessing from an old list of bot names is not a substitute for checking current vendor documentation and verified logs.

The homepage needs the same restraint. It receives plenty of checks, and its verified share sits near one in five, but the logs do not show it performing like the guides as a source for live answers. Scale makes the traffic feel persuasive: an industry account reports roughly [50 billion AI crawler requests a day in March 2025, while Googlebot still reached more URLs than ClaudeBot or PerplexityBot](https://nohacks.co/blog/ai-user-agents-landscape-2026). Those are ecosystem figures, not a multiplier for our site. They help explain why a busy server log and a thin stream of useful retrieval can coexist.

**Keep access working, but do not mistake access checks for demand for the homepage.**

## The decision: fix the measurement, then fix one page

If I were looking at this on a business site, I would change the report before changing the content. The order matters:

1. **Verify claimed bots by IP.** Keep the vendor lists current. A user-agent-only trend is not a reliable starting point.
2. **Split requests by job.** Separate training crawls, search indexing, and user-triggered retrieval instead of presenting one crawler total.
3. **Compare pages using matching denominators.** For pages with both figures, compare verified fetches with fetches that fed live answers. Mark missing counts as missing rather than filling the gap with a claimed-hit total.
4. **Choose one narrow guide to improve.** Give it a question it can answer directly, a date that reflects real maintenance, and numbers, definitions, or steps that do the work on the page. Then watch the same measures over time.

Crawl access is part of that work, but the controls are not interchangeable. If the aim is to keep OpenAI’s training crawler out while leaving its search and user-request paths open, the [documented distinction among GPTBot, OAI-SearchBot, and ChatGPT-User](https://www.honeyb.ai/blog/gptbot-crawler-guide) matters. Check the rules for each service rather than copying an OpenAI rule onto Perplexity or Claude. A broad block may be simpler to type and harder to diagnose later.

The content decision is narrower than “publish more AI-friendly pages.” Our YouTube guide did not stand out because the logs show it was long, and the logs cannot tell us that a date alone made it useful. It stood out because verified agents fetched it and it repeatedly fed live answers, while category pages with thousands of claimed hits fed none. I would use that contrast to inspect the guide’s structure and update another page that can answer a similarly specific question. I would not turn a category index into a longer category index and call it a test.

## The limits: one site, top paths, 30 days

This is **one site with an ads-focused audience**, observed for 30 days. The path list available for this analysis was truncated to top paths. One-off fetches in the long tail are missing, so I cannot say these are the only guides that fed answers. I also cannot turn a correlation between page shape and live-answer fetches into proof that formatting alone caused the difference.

The window is long enough to expose a stark contrast in this site’s logs, but the exact counts can move. The outside tracking synthesis reports [day-to-day source churn](https://www.linkedin.com/pulse/heres-what-ai-actually-looks-before-citing-your-content-rohit-rajpal-qaxbf); a different month could change which guides appear in the table. Treat our rates as a prompt to examine your own pages, not benchmarks to paste into a forecast.

There is a more basic limit, too: **a fetch that fed an answer is not a record of a visible citation.** The logs support a decision about which pages and requests to investigate. They do not let me promise that every answer displayed a link, or that making another page look like the YouTube guide will reproduce its rate.

The decision they _do_ support is clear. Verify bot traffic before reporting it. Separate live retrieval from the other work bots do. Then put the next content effort into a page that can answer a specific question with current, extractable details. If a page has no fresh numbers, definitions, or steps to offer, I would skip the cosmetic rewrite. It may attract another hit without giving an answer anything useful to carry.

Put the month in one picture and it looks like a funnel, not a flood. Thousands of claimed hits arrive at the top. IP checks remove most homepage claims. Bot roles separate background activity from user-triggered retrieval. At the bottom, a dated guide feeds answers while category pages with heavy claimed traffic feed none. That is why I stopped reporting crawls as visibility.

![Funnel from claimed AI bot hits through IP verification to live-answer fetches](https://groas.com/media/blog/b628de43a9be62e1f9cea8508bec7142f78fae8dc09e8068a08ec3339f3b9611.png)

The pages that made it through looked nothing like a homepage or a category index. They looked like work: a year in the title, a date on the page, numbers and steps an answer could use without guessing. **Build that shape once, then let verified logs tell you which question the page actually answers.**

![Dated 2026 guide beside a chat answer](https://groas.com/media/blog/3a6ed68bc60910edfd3c3ee6f9d766d9dbc7a4ab86d43564f68ae0ec68dbcb56.png)

## Frequently Asked Questions

### How much of the AI bot traffic in server logs is actually genuine?

On the groas.com homepage, 11,441 hits in 30 days claimed an AI bot user-agent, but only 2,187 survived IP verification against vendor lists, a verified share of 19.1%. That means roughly four out of five claimed AI hits did not pass the check, so an unchecked user-agent chart should not be treated as evidence of AI attention.

### Why did category pages get thousands of AI bot hits but no citations in ChatGPT answers?

The category pages each drew more than 3,000 claimed bot hits over 30 days, yet none fed a live ChatGPT, Claude, or Perplexity answer in the dataset. A hit only shows something requested a URL; it does not show the request came from a genuine agent gathering material for a user question. The pages also list routes to other pages rather than offering specific, extractable answers.

### How can I tell real AI crawler traffic from spoofed traffic?

Check the request IP against each vendor's published list rather than trusting the user-agent string, since any script can claim to be ChatGPT-User. One outside two-week sample across 16 AI crawlers found 5.7% of traffic claiming an AI crawler identity was spoofed, with about one in six requests claiming to be ChatGPT-User fake. Keep the vendor lists current on a refresh schedule.

### What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?

OpenAI uses separate robots.txt tokens for each: GPTBot crawls for training, OAI-SearchBot handles search indexing, and ChatGPT-User makes a page pull related to a user request. Blocking one is not the same as blocking the others, so counting all bots in one total hides decisions that affect different parts of the path to an answer.

### What made the YouTube ads guide get cited in AI answers while other pages did not?

The YouTube ads 2026 guide had 294 verified fetches, 273 of which fed a live answer, or 92.9% in the 30-day slice. It names a specific subject, carries a visible date, and puts numbers, steps, and definitions in the body, so a retrieval agent gets an answer it can use. Category pages only list links and ask the agent to keep looking.

### Does having a visible last-updated date on a page help with AI citations?

Possibly, but not as a guarantee. Outside analyses report that 95% of cited pages had been updated within 10 months and pages with a visible last-updated date earned 1.8 times more citations. Treat the date as a prompt to refresh the numbers and steps that actually changed, not as a cosmetic stamp, since a fetch that fed an answer is not a record of a visible citation.

### Should robots.txt requests from AI bots be counted as citations?

No. The logs showed 1,160 Claude-User requests classified as live-answer fetches hitting the robots.txt file, but nobody needs a quote from that file in a chat answer. Use those requests to check that the file is available, current, and explicit about which agents are allowed, and keep the rules for each service distinct rather than copying an OpenAI rule onto Perplexity or Claude.

## Related Posts

- [![Clay figure pours a wheelbarrow of articles into a wall funnel whose pipe is disconnected, so the pages pile on the floor beside an unused wrench.](https://groas.com/media/blog/e1a99963f159a4756e614472ee73400643c5900fa7bde718abe8fa563e8cdc77.png)](https://groas.com/post/can-chatgpt-even-read-your-site-a-45-min)
  ### [Can ChatGPT Read Your Site? A 45-Minute Check Before You Write More AEO Content](https://groas.com/post/can-chatgpt-even-read-your-site-a-45-min)
  [October 6, 2026 • 10 min read](https://groas.com/post/can-chatgpt-even-read-your-site-a-45-min)

  [Written by Alexander Perelman](https://groas.com/post/can-chatgpt-even-read-your-site-a-45-min)
- [![Cartoon of a marketer noting 'position two' after one pull of a slot machine whose reels are already spinning to a new result.](https://groas.com/media/blog/038bfefb4237637a1718ff66b379d5bdc42d57615f25797529aa5ef5682a4f7e.png)](https://groas.com/post/i-measured-ai-visibility-like-a-ppc-acco)
  ### [Six PPC Habits That Misread AI Visibility](https://groas.com/post/i-measured-ai-visibility-like-a-ppc-acco)
  [October 6, 2026 • 10 min read](https://groas.com/post/i-measured-ai-visibility-like-a-ppc-acco)

  [Written by David](https://groas.com/post/i-measured-ai-visibility-like-a-ppc-acco)
- [![A tiny worker on a ladder fixes up five model shopfronts while a giant dusty 'take a number' dispenser looms behind, its ticket queue ignored.](https://groas.com/media/blog/35c8ce6c06242ab08178711c5abe08dda0df51ec927a2dd6bc9770aa5f83bdbf.png)](https://groas.com/post/one-strategist-zero-developers-five-clie)
  ### [One Strategist, Zero Developers, Five Domains: A 30-Day AEO Launch Playbook](https://groas.com/post/one-strategist-zero-developers-five-clie)
  [October 6, 2026 • 10 min read](https://groas.com/post/one-strategist-zero-developers-five-clie)

  [Written by David](https://groas.com/post/one-strategist-zero-developers-five-clie)

## The Machines Already Run Search, You Should Probably Own One

[get started free](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#webpage","url":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our","name":"30 Days of AI Bot Logs: Thousands of Hits, Zero Answers for Category Pages","description":"Our homepage logged 11,441 hits claiming to be AI bots in 30 days. Only 2,187 were verified.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://groas.com/media/blog/82ca40907f165aa6ef6baa0903a5a70019e13bb782846dbcacfd92d25485cc72.png"},"breadcrumb":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#breadcrumb"},"dateModified":"2026-10-06T05:33:23.349Z","mainEntity":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"AI For Google Ads","item":"https://groas.com/category/ai-for-google-ads"},{"@type":"ListItem","position":4,"name":"30 Days of AI Bot Logs: Thousands of Hits, Zero Answers for Category Pages","item":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#article","url":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our","mainEntityOfPage":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"30 Days of AI Bot Logs: Thousands of Hits, Zero Answers for Category Pages","description":"Our homepage logged 11,441 hits claiming to be AI bots in 30 days. Only 2,187 were verified. I used to tell clients to watch bot hits in the logs as a proxy for AI visibility. I was wrong. The rest of the month made the distinction harde…","datePublished":"2026-10-06T05:33:22.844Z","dateModified":"2026-10-06T05:33:23.349Z","image":{"@type":"ImageObject","url":"https://groas.com/media/blog/82ca40907f165aa6ef6baa0903a5a70019e13bb782846dbcacfd92d25485cc72.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"articleSection":"AI For Google Ads","keywords":"Google Ads, AI For Google Ads","wordCount":2197,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our#faq","mainEntity":[{"@type":"Question","name":"How much of the AI bot traffic in server logs is actually genuine?","acceptedAnswer":{"@type":"Answer","text":"On the groas.com homepage, 11,441 hits in 30 days claimed an AI bot user-agent, but only 2,187 survived IP verification against vendor lists, a verified share of 19.1%. That means roughly four out of five claimed AI hits did not pass the check, so an unchecked user-agent chart should not be treated as evidence of AI attention."}},{"@type":"Question","name":"Why did category pages get thousands of AI bot hits but no citations in ChatGPT answers?","acceptedAnswer":{"@type":"Answer","text":"The category pages each drew more than 3,000 claimed bot hits over 30 days, yet none fed a live ChatGPT, Claude, or Perplexity answer in the dataset. A hit only shows something requested a URL; it does not show the request came from a genuine agent gathering material for a user question. The pages also list routes to other pages rather than offering specific, extractable answers."}},{"@type":"Question","name":"How can I tell real AI crawler traffic from spoofed traffic?","acceptedAnswer":{"@type":"Answer","text":"Check the request IP against each vendor's published list rather than trusting the user-agent string, since any script can claim to be ChatGPT-User. One outside two-week sample across 16 AI crawlers found 5.7% of traffic claiming an AI crawler identity was spoofed, with about one in six requests claiming to be ChatGPT-User fake. Keep the vendor lists current on a refresh schedule."}},{"@type":"Question","name":"What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?","acceptedAnswer":{"@type":"Answer","text":"OpenAI uses separate robots.txt tokens for each: GPTBot crawls for training, OAI-SearchBot handles search indexing, and ChatGPT-User makes a page pull related to a user request. Blocking one is not the same as blocking the others, so counting all bots in one total hides decisions that affect different parts of the path to an answer."}},{"@type":"Question","name":"What made the YouTube ads guide get cited in AI answers while other pages did not?","acceptedAnswer":{"@type":"Answer","text":"The YouTube ads 2026 guide had 294 verified fetches, 273 of which fed a live answer, or 92.9% in the 30-day slice. It names a specific subject, carries a visible date, and puts numbers, steps, and definitions in the body, so a retrieval agent gets an answer it can use. Category pages only list links and ask the agent to keep looking."}},{"@type":"Question","name":"Does having a visible last-updated date on a page help with AI citations?","acceptedAnswer":{"@type":"Answer","text":"Possibly, but not as a guarantee. Outside analyses report that 95% of cited pages had been updated within 10 months and pages with a visible last-updated date earned 1.8 times more citations. Treat the date as a prompt to refresh the numbers and steps that actually changed, not as a cosmetic stamp, since a fetch that fed an answer is not a record of a visible citation."}},{"@type":"Question","name":"Should robots.txt requests from AI bots be counted as citations?","acceptedAnswer":{"@type":"Answer","text":"No. The logs showed 1,160 Claude-User requests classified as live-answer fetches hitting the robots.txt file, but nobody needs a quote from that file in a chat answer. Use those requests to check that the file is available, current, and explicit about which agents are allowed, and keep the rules for each service distinct rather than copying an OpenAI rule onto Perplexity or Claude."}}]}]}
```