---
title: "905 Live-Answer Fetches in 30 Days: What Our AI Bot Logs Say About Dated Pages"
description: "In 30 days of groas edge logs, two dated changelog posts drew 905 verified live-answer fetches. Here is what the counts show, what they cannot prove, and why raw bot hits are a poor buying metric."
url: "https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ab69b9fd-2b6f-4465-854d-b4eccd690644.png"
published: "2026-11-04T05:53:56.654Z"
modified: "2026-11-10T21:25:23.408Z"
---

November 4, 2026 · 9 min read

# 905 Live-Answer Fetches in 30 Days: What Our AI Bot Logs Say About Dated Pages

[](https://groas.com/author/alexander-perelman)

[Alexander Perelman](https://groas.com/author/alexander-perelman)Personally Tested

Head Of Product @ groas

[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

Updated November 10, 2026

![A paper model town where a long queue of tiny couriers lines up at a kiosk made of a calendar page while a glossy brochure headquarters stands almost empty.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ab69b9fd-2b6f-4465-854d-b4eccd690644.png)

In this article

1. [The dataset: what we counted and what we cannot see](#the-dataset-what-we-counted-and-what-we-cannot-see)
2. [Finding 1: dated changelogs drew the clearest content-page demand](#finding-1-dated-changelogs-drew-the-clearest-content-page-demand)
3. [Finding 2: live-answer bots do not all visit the same paths](#finding-2-live-answer-bots-do-not-all-visit-the-same-paths)
4. [Finding 3: raw media hits make a poor visibility report](#finding-3-raw-media-hits-make-a-poor-visibility-report)
5. [Finding 4: keep the commercial page, but give it the right job](#finding-4-keep-the-commercial-page-but-give-it-the-right-job)
6. [What I would change in the content budget](#what-i-would-change-in-the-content-budget)
7. [Check robots.txt before paying for another page](#check-robotstxt-before-paying-for-another-page)

Two dated Google Ads changelog posts on our site drew [905 verified live-answer fetches in 30 days](https://groas.com/post/30-days-of-ai-crawler-logs-two-dated-changelog-posts-got-53). Our `/for-businesses` page drew 68. If you are paying for AI search visibility, that gap points toward pages that answer a dated question, not another round of generic service copy.

It does **not** mean those posts appeared in 905 answers. A fetch tells us an assistant pulled a page while responding to a user; it does not tell us whether the answer cited it, named groas, or sent us a customer. That distinction is the point. I used to dismiss bot traffic reports because they counted everything. These edge logs let us separate verified live-answer fetches from background crawling and user agents merely claiming to be bots. Once you make that split, the pages worth studying look different from the pages with the loudest raw-hit totals.

## The dataset: what we counted and what we cannot see

This is [30 measured days of edge-level logs from groas.com](https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-on-o). **Verified** means the requesting IP matched the operator’s published list for that bot. **Unverified** means a request claimed a bot identity in its user agent but failed that check. **Live-answer fetch** means a user-triggered bot requested a page while answering a question, rather than a background crawler visiting it for later indexing.

Those labels are not interchangeable. A user agent is easy to claim; [HUMAN Security found about 5.7% of requests claiming to be AI crawlers were spoofed](https://deepsmith.ai/blog/verify-identify-real-ai-crawler-traffic). And when a CDN sits between a visitor and the site, the origin sees the CDN address. Verification needs [edge-level signals](https://www.siteauditlint.com/blog/cloudflare-ai-crawler-monitoring), not an origin log interpreted as though it recorded the visitor’s IP.

**Free tool:** the [AI Crawler Checker](https://groas.com/free-tools/ai-crawler-checker) shows which AI crawlers your robots.txt lets in and what they receive when they arrive.

There is another limit: the path list available for this analysis is truncated. A URL missing from the visible rows did not necessarily receive zero fetches. I cannot rank every page on the site, or say these changelog posts were the sitewide winners. I can compare the visible paths with verified counts, show where raw traffic becomes misleading, and say what I would inspect before buying more content.

That is enough to make a decision. It is not enough to declare a universal citation rate.

## Finding 1: dated changelogs drew the clearest content-page demand

The visible content-page contrast is sharp:

| Page                                   | Verified live-answer fetches in the measured period |
| -------------------------------------- | --------------------------------------------------: |
| Dated Google Ads changelog post        |                                                 513 |
| Second dated Google Ads changelog post |                                                 392 |
| **Both posts combined**                |                                             **905** |
| `/for-businesses`                      |                                                  68 |

The posts were about what changed in Google Ads and when. That gives an assistant a dated fact to retrieve when someone asks a dated question. A service page has a different job: it explains who we serve and what we do. I would not delete it because it drew fewer fetches. I would stop expecting it to answer a question about a recent change.

Outside this dataset, [one analysis found that 76.4% of Perplexity’s top-cited pages had been updated within 30 days](https://clearadsagency.com/89-of-what-works-on-chatgpt-doesnt-work-on-perplexity/). That finding offers a plausible reason to take freshness seriously. It does not identify which assistant fetched each of our posts, or prove that freshness alone caused the difference. Topic, existing visibility, and the questions people asked also matter; our visible path counts do not separate those effects.

Still, the publishing implication is practical. If a buyer asks what changed this month, give the assistant a page that says what changed this month. Put the date and the change where a reader can find them. **Make the answer easier to retrieve than the sales pitch.**

## Finding 2: live-answer bots do not all visit the same paths

The bot-and-path rows show why I would not roll every verified request into one grand “AI visibility” number:

| Bot and path               |                                       Verified live-answer fetches in the measured period | What the request shows                                     |
| -------------------------- | ----------------------------------------------------------------------------------------: | ---------------------------------------------------------- |
| ChatGPT-User, homepage `/` |                                                                                       789 | A live request for the homepage                            |
| Claude-User, `/robots.txt` | [890](https://groas.com/post/claude-checked-our-robots-txt-1-071-times-in-30-days-the-ai) | Repeated requests for access rules                         |
| Claude-User, homepage `/`  |                                                                                        35 | Fewer live homepage requests than permission-file requests |

ChatGPT-User’s homepage requests are not evidence that all those users asked about groas by name. The logs show the page requested, not the wording of the prompt. Likewise, Claude’s requests for `/robots.txt` are not citations. They show that access rules featured heavily in the paths we could see for that bot. **A permission-file fetch is not a content win.**

The distinction also matters when you set access policy. [ChatGPT-User is a user-triggered fetcher, while OAI-SearchBot has a separate search-crawling role](https://aicrawlercheck.com/blog/chatgpt-user-openai-search-crawler). The same source says OpenAI’s robots.txt rules may not apply to ChatGPT-User requests. [Anthropic’s Claude bots honor robots.txt](https://agentsurface.dev/docs/discovery/robots-txt). One rule of thumb for “AI bots” will not describe both behaviors.

I learned the danger of universal rules in paid search, usually after a neat account-wide assumption met a messy account. Here the check is simpler: inspect which bot requested which path, then inspect what your access rules allow. Do that before treating a homepage fetch or a robots.txt fetch as a recommendation.

## Finding 3: raw media hits make a poor visibility report

The asset paths are where a raw-hit dashboard starts to flatter itself:

| Visible asset path group |                               Raw hits |    Verified share |
| ------------------------ | -------------------------------------: | ----------------: |
| One image path           |                                  2,395 |      282 verified |
| Icon paths               | Not reported in the visible comparison | Under 5% verified |

I cannot label every unverified request a spoof. Some may be other automated traffic; the visible counts do not resolve each request’s purpose. What I can say is that **raw hits on an image are not evidence that an assistant recommended the business**. Even a verified asset request is not, by itself, an answer containing your brand.

A separate [Cloudflare example recorded 897,000 AI requests, with a `.webp` blog image receiving 35,190 requests](https://www.siteauditlint.com/blog/cloudflare-ai-crawler-monitoring). That is a useful warning about media volume, not a benchmark for our site. It shows how easily an asset can dominate a request chart while telling you little about what a person saw in an answer.

There is a second narrowing step after retrieval: citation. In [one measured set, about 110 of 3,554 retrieved pages appeared in the answer](https://superpath.com/blog/retrieved-vs-cited-the-two-funnel-model-of-ai-visibility-for-home-service-businesses/). Those are not our citation counts, and I would not apply that rate to our posts. The mechanism is the useful part: a system can fetch a page, inspect it, and leave it out of the final response.

So ask for the report in layers: raw requests, verified requests, live-answer fetches, and any separately measured citations or outcomes. If a vendor hands you the first layer and calls it demand, they are counting plumbing and calling it preference.

## Finding 4: keep the commercial page, but give it the right job

The `/for-businesses` page received 68 verified live-answer fetches. That is less than either dated post in the visible rows, but it is not zero. A commercial page can answer a question about a company’s offer. A changelog answers a question about a change. They compete for different tasks, not for the right to exist.

I used to tell clients that one strong service page could carry most of this work. I was wrong to give it that much responsibility. It needs clear positioning that an assistant can understand. It cannot also be a current account of every rule, control, or cost that moved in the category. Stuffing a service page with updates would make both jobs harder.

An [outside analysis reports that 87% of ChatGPT citations in its sample came from Bing top results](https://clearadsagency.com/89-of-what-works-on-chatgpt-doesnt-work-on-perplexity/), while its Perplexity findings emphasize more recent pages. That helps explain why one publishing approach may not serve every search surface equally. It does **not** let me assign our homepage fetches to ChatGPT prompts or our changelog fetches to Perplexity prompts. Our path table does not contain that answer.

The decision is narrower and more useful: keep the commercial page accurate and easy to read, then publish separate dated pages when there is a specific change worth answering. Do not make one URL pretend to be both a company introduction and a running change log.

## What I would change in the content budget

I would fund **pages that answer a dated, specific question** before buying more undifferentiated service copy. The two visible changelog posts gave us a reason to test that priority; they did not give us a guaranteed fetch count for the next post. The writing brief is straightforward: identify the change, put its date in the title, and state the useful fact early. The draft advice to put it in the first 100 words is a good discipline, not a rule the logs can prove.

For a business outside paid search, the equivalent might be a change in its category, a cost that moved, or a rule that now affects buyers. Publish when there is something to say. A date on a page with no meaningful change is decoration, and an assistant does not owe it a fetch.

I would also change the reporting brief. Ask anyone selling AI search visibility for:

- **Verified requests by page**, with the bot identity checked against the relevant operator list rather than accepted from the user agent.
- **Live-answer fetches separated from background crawling**, so a content request is not buried beside routine indexing.
- **Asset paths separated from content paths**, so images and icons cannot inflate a page-performance claim.
- **Citations and business outcomes reported separately**, where they can actually be measured, rather than inferred from a fetch.

[OpenAI publishes separate IP lists for its bots](https://aicrawlercheck.com/blog/chatgpt-user-openai-search-crawler). Those lists can change, so a report based on a neglected hard-coded allowlist can drift. Behind a CDN, the verification needs to use edge data rather than assuming the origin recorded the requesting bot’s IP. If a vendor cannot explain that distinction, I would not let its raw-hit chart decide the next content retainer.

## Check robots.txt before paying for another page

Claude-User requested our `/robots.txt` 890 times in the visible live-answer rows. That count is not a reason to optimize the permission file for “engagement.” It is a reason to make sure the pages you want read are accessible to the crawlers that honor it.

Open the file. Check whether it allows the blog and changelog paths you want available. Check for an accidental disallow on the pages you are paying to publish. Then check the policy for the bot whose visibility you care about: [OAI-SearchBot and ChatGPT-User do different work](https://aicrawlercheck.com/blog/chatgpt-user-openai-search-crawler), so a ChatGPT-User fetch alone does not establish that the search crawler can reach the page. The practical order is **access first, content second, measurement third**. Paying for writing while blocking the intended reader is expensive comedy.

The same caution applies to the report you get back. On Cloudflare Free, [crawler identification described in this monitoring example relies on user-agent strings, while paid plans offer additional bot-detection signals](https://www.siteauditlint.com/blog/cloudflare-ai-crawler-monitoring). A claimed bot name in a report is not the verification standard used in this analysis. Ask which signal the report used before you treat a request as a confirmed visit.

This is the work I would expect from a company managing AI search visibility on an ongoing basis: keep the right pages reachable, publish dated answers when the subject warrants them, and show verified live-answer requests by page without pretending those requests are citations. [groas offers SEO and AI Search work](https://groas.com/pricing) alongside technical fixes and new pages, with attention to downstream revenue rather than raw bot counts. The report should make the difference between those measures plain.

Thirty days on one site cannot tell us which prompts lead to citations across ChatGPT, Perplexity, and Gemini in general. A truncated path list cannot establish a sitewide winner. But these logs do support a spending decision: inspect access, separate verified live-answer fetches from noise, and give dated questions their own pages. **Run that split on your domain before you approve another content retainer.**

## Frequently Asked Questions

### Why did dated changelog posts draw more live-answer fetches than service pages?

Dated changelog posts drew 905 verified live-answer fetches in 30 days, whereas the service page /for-businesses received only 68 fetches. This difference suggests that content addressing dated questions, such as recent changes, performs better with AI assistants.

### Are raw hits on images or media assets a reliable measure of AI visibility?

Raw hits on an image are not evidence that an assistant recommended the business. Even a verified asset request is not, by itself, an answer containing the business brand. A comprehensive report should layer raw requests with verified requests, live-answer fetches, and actual citations.

### What is the difference between a bot fetching a homepage versus fetching a robots.txt file?

Bot requests for various paths, such as a homepage or a permission file, do not automatically equate to content citation or a visibility win. For example, Claude-User requested the /robots.txt file 890 times, which indicates access rules were scanned, not that the content was utilized.

### How is AI bot traffic verified in an edge log analysis?

AI bot traffic must be verified by matching the requesting IP against the operator’s published list for that bot. This verified status, known as a live-answer fetch, differs from unverified requests that only claim a bot identity in their user agent.

### Should service pages be updated with regular business changes or specific product updates?

A service page should maintain clear positioning about who the company serves and what it does, rather than acting as a running account of every rule or cost change. Publishing separate dated pages is the recommended approach for specific, temporal updates.

### What technical steps should be taken before relying on bot traffic data for content decisions?

Before making content decisions based on visibility, businesses must check their robots.txt file for disallows on target paths. It is necessary to identify which specific bots, such as OAI-SearchBot or ChatGPT-User, apply to the visibility goals.

## Run Your Marketing On groas

Plug ChatGPT, Claude or any AI agent into groas and let it do the work of a whole marketing team: free to ask, $49 a month flat with every feature included, and autopilot whenever you’re ready.

[Start Free](https://groas.com/pricing)

[Or have groas run it for you](https://groas.com/done-for-you)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is the marketing infrastructure for your own AI: plug Claude, ChatGPT or any AI agent into groas and run all your marketing from it, for a tiny fraction of the usual cost, with the groas app there whenever you want to see more. It is the layer ChatGPT, Claude and any AI assistant run a business’s online marketing on. Through groas an assistant can do anything you could do natively in Google Ads, Bing Ads, Meta Ads and ChatGPT ads, on your website and in AI search, and, where the owner switches it on, can run connected campaigns by itself, so AI agents do the work a whole marketing team used to (free to read, $49 a month to make changes, delivered as an MCP server). Separately, its done-for-you service has senior people and proprietary quantitative systems run paid ads, SEO and AI search for businesses, and white label for agencies, from $5,000 a month.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#webpage","url":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two","name":"905 Live-Answer Fetches in 30 Days: What Our AI Bot Logs Say About Dated Pages","description":"In 30 days of groas edge logs, two dated changelog posts drew 905 verified live-answer fetches. Here is what the counts show, what they cannot prove, and why raw bot hits are a poor buying metric.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ab69b9fd-2b6f-4465-854d-b4eccd690644.png"},"breadcrumb":{"@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#breadcrumb"},"dateModified":"2026-11-10T21:25:23.408Z","mainEntity":{"@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"905 Live-Answer Fetches in 30 Days: What Our AI Bot Logs Say About Dated Pages","item":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#article","url":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two","mainEntityOfPage":{"@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"905 Live-Answer Fetches in 30 Days: What Our AI Bot Logs Say About Dated Pages","description":"In 30 days of groas edge logs, two dated changelog posts drew 905 verified live-answer fetches. Here is what the counts show, what they cannot prove, and why raw bot hits are a poor buying metric.","datePublished":"2026-11-04T05:53:56.654Z","dateModified":"2026-11-10T21:25:23.408Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ab69b9fd-2b6f-4465-854d-b4eccd690644.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"live-answer fetches, fetches, live-answer, bot, dated, page, about, pages","wordCount":1980,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/what-30-days-of-ai-bot-traffic-on-our-own-site-showed-two#faq","mainEntity":[{"@type":"Question","name":"Why did dated changelog posts draw more live-answer fetches than service pages?","acceptedAnswer":{"@type":"Answer","text":"Dated changelog posts drew 905 verified live-answer fetches in 30 days, whereas the service page /for-businesses received only 68 fetches. This difference suggests that content addressing dated questions, such as recent changes, performs better with AI assistants."}},{"@type":"Question","name":"Are raw hits on images or media assets a reliable measure of AI visibility?","acceptedAnswer":{"@type":"Answer","text":"Raw hits on an image are not evidence that an assistant recommended the business. Even a verified asset request is not, by itself, an answer containing the business brand. A comprehensive report should layer raw requests with verified requests, live-answer fetches, and actual citations."}},{"@type":"Question","name":"What is the difference between a bot fetching a homepage versus fetching a robots.txt file?","acceptedAnswer":{"@type":"Answer","text":"Bot requests for various paths, such as a homepage or a permission file, do not automatically equate to content citation or a visibility win. For example, Claude-User requested the /robots.txt file 890 times, which indicates access rules were scanned, not that the content was utilized."}},{"@type":"Question","name":"How is AI bot traffic verified in an edge log analysis?","acceptedAnswer":{"@type":"Answer","text":"AI bot traffic must be verified by matching the requesting IP against the operator’s published list for that bot. This verified status, known as a live-answer fetch, differs from unverified requests that only claim a bot identity in their user agent."}},{"@type":"Question","name":"Should service pages be updated with regular business changes or specific product updates?","acceptedAnswer":{"@type":"Answer","text":"A service page should maintain clear positioning about who the company serves and what it does, rather than acting as a running account of every rule or cost change. Publishing separate dated pages is the recommended approach for specific, temporal updates."}},{"@type":"Question","name":"What technical steps should be taken before relying on bot traffic data for content decisions?","acceptedAnswer":{"@type":"Answer","text":"Before making content decisions based on visibility, businesses must check their robots.txt file for disallows on target paths. It is necessary to identify which specific bots, such as OAI-SearchBot or ChatGPT-User, apply to the visibility goals."}}]}]}
```
