---
title: "Our AI Bot Logs Had 14,742 robots.txt Hits. Only 3,894 Were Verified."
description: "A 30-day look at groas.com edge logs shows why claimed bot hits mislead, where verified live-answer fetches concentrated, and what agencies should report instead."
url: "https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2bc7480c-4fac-4cfb-be36-1f12c3917884.png"
published: "2026-10-26T05:26:15.225Z"
modified: "2026-10-26T05:26:15.303Z"
---

October 26, 2026 · 10 min read

# Our AI Bot Logs Had 14,742 robots.txt Hits. Only 3,894 Were Verified.

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![A bouncer at a night-time door checks a long queue of shady figures all wearing 'ChatGPT-User' name tags, letting only one in, because a bot label is just a claim.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2bc7480c-4fac-4cfb-be36-1f12c3917884.png)

In this article

1. [What our edge logs can—and cannot—count](#what-our-edge-logs-canand-cannotcount)
2. [Finding 1: the raw bot count shrinks under verification](#finding-1-the-raw-bot-count-shrinks-under-verification)
3. [Finding 2: live fetches favour a few pages](#finding-2-live-fetches-favour-a-few-pages)
4. [Finding 3: the fetcher matters as much as the page](#finding-3-the-fetcher-matters-as-much-as-the-page)
5. [Why a prompt-rank chart is not the answer](#why-a-prompt-rank-chart-is-not-the-answer)
6. [The agency-tool trial checklist](#the-agency-tool-trial-checklist)
7. [What this month does not prove](#what-this-month-does-not-prove)

Only **3,894 of 14,742** requests to our `robots.txt` file that claimed to be AI crawlers passed an IP verification check. If an agency had reported the raw count as AI attention, roughly three quarters of its headline number would have been unverified.

We logged 30 days of requests at the edge on groas.com. The more useful finding came after verification: two timely Google Ads changelog posts drew **922 live-answer fetches between them**, while a teardown post with 309 claimed bot hits drew just one. Those are fetches, not confirmed citations or answers that displayed our pages. But they make the reporting decision fairly clear. If I were choosing an AEO tool for an agency, I would ask for **verified live-answer fetches per client page**, not another prompt-rank chart.

## What our edge logs can—and cannot—count

I used to tell clients that bot traffic meant interest. I was wrong. A user-agent string is a claim made by the requester; it is not proof of identity. For this analysis, we logged requests on groas.com whose user agents claimed to be AI bots, then checked their source IPs against the operators’ published ranges. That distinction between a claimed bot and a verified one is the [verification rule](https://digimirror.io/your-bot-traffic-logs-are-lying-to-you/) behind every table here.

The source for our site figures is **our own 30-day edge-log window**. These are request counts, not analytics sessions, people, prompts or citations. The IP check tells us whether a request came from a published operator range. It does not tell us what a person asked, whether the fetched page appeared in an answer, or how often that answer was seen.

We then separated verified requests by job. Operators use [different bot classes](https://www.airanklab.com/blog/ai-crawlers-list-user-agents): training crawlers such as `GPTBot` and `ClaudeBot`, search or index crawlers such as `OAI-SearchBot` and `PerplexityBot`, and user-triggered fetchers such as `ChatGPT-User` and `Claude-User`. A request from the last group is the closest log-level signal here that a page was pulled for a live answer. It is still a fetch, not proof that the answer cited the page.

| Measure             | What it counts                                                   | What it does not establish                    |
| ------------------- | ---------------------------------------------------------------- | --------------------------------------------- |
| Claimed hits        | Requests with an AI-bot user agent                               | That the named operator sent them             |
| Verified hits       | Claimed requests whose source IP passed the operator-range check | That a person saw or cited the page           |
| Live-answer fetches | Verified requests from user-triggered fetchers                   | That the page appeared in the finished answer |

That is the order to read the figures below: **identity first, purpose second, citation separately**. Server logs can show a fetch, and a `200` response means the page was available to read; they cannot establish a citation on their own ([what server logs prove](https://www.scotttischler.com/articles/ai-crawler-server-logs)).

![Edge-log diagram separating claimed AI bot hits from IP-verified requests](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/8ec8c245-aac2-4f07-aa28-8c2bc9d62560.png)

## Finding 1: the raw bot count shrinks under verification

The largest claimed count appeared on `robots.txt`: **14,742 requests in 30 days**. Of those, **3,894 verified**, a 26.4% share. Put another way, 73.6% of the requests claiming an AI-crawler identity did not pass our check. Image requests were less convincing still: **87 of 1,271** verified, or 6.8%.

| Path          | Claimed hits | Verified hits | Verified share |
| ------------- | -----------: | ------------: | -------------: |
| `/robots.txt` |       14,742 |         3,894 |      **26.4%** |
| Image files   |        1,271 |            87 |       **6.8%** |

_Source: groas.com edge logs, one 30-day window. “Verified” means the source IP passed our check against published operator ranges; a request that failed the check is not, by itself, proof of who sent it._

The mechanism is less mysterious than the dashboard makes it look. Anyone can send a request bearing `ChatGPT-User` as its user agent. Checking the source IP changes the question from “What did this request call itself?” to “Did it come from the operator it named?” An [independent log analysis](https://digimirror.io/your-bot-traffic-logs-are-lying-to-you/) found a similar distinction in a smaller sample: six of 33 claimed live fetches verified, while some unverified requests sought credential files rather than page content. Those were not our requests. The comparison illustrates why a bot label alone is weak evidence, not why any particular request in our logs failed.

**Practical takeaway:** if a report leads with raw AI-bot hits, ask how many passed an IP check before calling the number attention.

![Bar chart comparing claimed and verified AI bot hits for robots.txt and image files](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/83c61d53-f98b-40fe-a5f1-051d4e8cb1ec.png)

## Finding 2: live fetches favour a few pages

Verification clears up _who_ made a request. The fetcher class helps explain _what kind_ of request it was. On groas.com, verified live-answer fetches concentrated on the homepage and two timely changelog posts. The teardown post shows why claimed traffic is a poor shortcut to that conclusion.

| Page             | Claimed hits | Verified hits |                        Live-answer fetches |
| ---------------- | -----------: | ------------: | -----------------------------------------: |
| Homepage         |      \~1,050 |         \~830 | **795**, including 751 from `ChatGPT-User` |
| Changelog post A |          868 |           845 |                                    **561** |
| Changelog post B |        \~480 |         \~460 |                                    **361** |
| Teardown post    |          309 |             6 |                                      **1** |

_Source: groas.com edge logs, the same 30-day window. Approximate counts are marked with `~`. Live-answer fetches are verified requests from user-triggered fetchers, not a count of published answers or citations._

The two changelog posts drew **561 and 361 live fetches**, respectively. Against their claimed hit totals, that is roughly 65% and 75%. One of them also had 845 verified requests out of 868 claimed, a sharp contrast with the `robots.txt` and image rows. The posts were timely answers to questions about Google Ads changes. The logs show that live fetchers repeatedly requested them; they do not show the prompts that led there or guarantee that the pages were cited.

Now read the teardown row from left to right. Its 309 claimed hits could have made it look busy in a raw-traffic report. Six passed verification. One was a live-answer fetch. If I had sorted the page report by claimed hits and stopped there, I would have given that post far more credit than these logs support.

**Practical takeaway:** put verified live-answer fetches beside each client URL. A sitewide bot total hides the page that was pulled for a live answer—and the page that merely attracted requests wearing a bot name.

## Finding 3: the fetcher matters as much as the page

Of the homepage’s **795 live-answer fetches**, **751 came from `ChatGPT-User`**. `Claude-User` appeared in a thinner pattern in this window: fewer pages and fewer repeats, with more activity on the homepage and the newer changelog post. That is a useful split for _our_ reporting, not a ranking of which engine matters most to every client.

It also explains why I would not roll every verified AI request into one “AI visits” line. A [training crawler, search crawler and user-triggered fetcher](https://www.airanklab.com/blog/ai-crawlers-list-user-agents) do different jobs. A verified training crawl says something different from a verified request made while a tool is assembling an answer. Both may matter to a site owner, but adding them together erases the distinction this analysis is trying to find.

Control is another reason to keep the classes separate. [User-triggered fetchers may treat `robots.txt` differently](https://www.airanklab.com/blog/ai-crawlers-list-user-agents) from crawlers doing routine collection, and `robots.txt` is not access control. If the decision is to block access, a robots directive should not be mistaken for enforcement; that calls for access rules such as WAF rules. Declared identity has limits, too: [Cloudflare’s account of de-listing Perplexity as verified](https://ppc.land/verified-bot/) describes both declared requests and an undeclared crawler rotating IPs past blocks. That account is context, not a measurement from groas.com.

**Practical takeaway:** report the page and the verified fetcher together. “AI bots visited” is too broad to tell a client whether live-answer systems pulled the work they paid to publish.

## Why a prompt-rank chart is not the answer

A prompt tracker can say a client ranked third for a set of prompts. It cannot, by itself, show whether a verified fetcher pulled the client’s page during the period being reported. Nor is one prompt run a stable view of citations. In [repeated ChatGPT runs of the same prompt](https://www.airanklab.com/blog/how-chatgpt-search-works), a median 7.7% of cited domains appeared in every run, with 0.54 overlap between consecutive runs. That is a reason to show repeated-run citation frequency rather than present one screenshot as a settled position.

Clicks are a separate limit on what a rank chart can promise. A [roundup of AI-search reporting data](https://wellows.com/blog/agency-reporting-for-ai-search/) cites Seer’s organic click-through change from 1.41% to 0.64% where AI Overviews appeared, and Pew’s 8% click rate with an AI summary versus 15% without one. Those figures do not measure our pages or prove the effect of any client’s citation. They do make it harder to treat placement in an answer as interchangeable with a visit.

I stopped sending rank screenshots as the main event. The more defensible report has two views side by side: **verified live-answer fetches by client page and fetcher**, then **citation frequency across repeated prompt runs**. The first says which pages the logs show being pulled for live answers. The second checks how often monitored prompts produced a citation. Neither substitutes for the other, and neither is closed-won revenue.

Our teardown post is the useful test case. A raw count gave it 309 claimed hits. Verification reduced that to six, and the live-fetch cut reduced it to one. A prompt dashboard on its own could not have shown that progression. Before paying for a chart, I want to know whether the tool can make those cuts.

## The agency-tool trial checklist

For an agency juggling multiple client accounts, a solo-brand prompt tracker can leave the actual reporting work in spreadsheets. An [agency-tool comparison](https://www.airanklab.com/blog/best-aeo-trackers-agencies-marketing-teams) lists multi-brand workspaces, white-label reporting, bulk prompt management, API access, per-brand pricing and competitor share of voice among the buying considerations. It also lists Otterly AI from $29 a month, Peec AI from $89, Profound Growth at $399 and Scrunch Agency Core at $500. Those are listed starting prices, not evidence that any of those tools passes the log checks below. Use a trial to test the output, not a sales call to imagine it.

1. **Claimed versus IP-verified hits for each client domain.** Ask to see the verification rule; a user-agent count alone can include unverified requests.
2. **Verified live-answer fetches per URL and fetcher.** Separate `ChatGPT-User`, `Claude-User` and comparable user-triggered fetchers from training and indexing crawlers.
3. **A page table that keeps claimed, verified and live counts together.** A live rate without its counts can conceal a page with almost no activity.
4. Engine split and competitor share of voice. Keep comparisons tied to named competitors and monitored prompts.
5. Month-over-month movement beside named fixes. Do not make the client infer a cause from a line moving on a chart.
6. **White-label export and bulk prompt control across client brands.** Otherwise the team assembles the report by hand.
7. **Repeated-run citations beside the fetch log.** Skip this pairing and you can mistake a fetched page for a cited one, or a single prompt result for a durable position.

## What this month does not prove

This is **one domain over 30 days**. Groas.com publishes in a niche where timely posts about Google Ads changes can match questions people are asking now. That helps explain why the changelog posts stood out, but the logs do not isolate freshness as the cause. Their roughly 65% and 75% live-fetch shares are not a forecast for an evergreen services page, another client, or next month.

The logs also do not prove that any of the 922 fetches for those two posts became a visible citation. A successful page response shows availability to read. An answer may still draw on other sources or cite none. This is why I would pair logs with repeated prompt checks rather than rename a fetch count “answers fed” and hope nobody asks what it means.

Finally, `ChatGPT-User` dominated the live-fetch pattern on our homepage in this window. A local services client might have a different engine split. **The method travels; these shares do not.** Verify identity, separate the fetcher’s job, and compare pages before drawing a client-level conclusion.

The buying decision is plain. Stop paying for reports that treat claimed bot hits as attention or a single prompt rank as a result. Ask for **verified live-answer fetches per client page, split by fetcher**, with repeated-run citation frequency alongside them. That is the strongest read-to-citation view these numbers support without pretending a fetch is a citation.

For agencies, groas pairs paid and organic search execution with [branded weekly reports](https://groas.com/for-agencies) under your name. Put any AEO reporting tool, including ours, through the same test: can it show what was claimed, what verified, what was pulled for a live answer, and what appeared in repeated citation checks? If it cannot show the fetch first, keep your money.

## Frequently Asked Questions

### How much AI bot traffic in server logs is actually verifiable?

On groas.com, only 3,894 of 14,742 requests to robots.txt that claimed an AI-crawler identity passed an IP verification check over 30 days, a 26.4% share. That means 73.6% of the claimed requests did not come from a published operator IP range.

### How do you verify that a request really came from an AI bot like ChatGPT or Claude?

Check the request's source IP against the operator's published IP ranges. A user-agent string is only a claim made by the requester, so verification changes the question from what the request called itself to whether it came from the operator it named.

### Which pages actually get fetched for live AI answers?

On groas.com, verified live-answer fetches concentrated on the homepage and two timely Google Ads changelog posts, which drew 561 and 361 fetches respectively, or 922 between them. The homepage accounted for 795 live-answer fetches, including 751 from ChatGPT-User.

### Can a page with lots of claimed bot hits have almost no real AI activity?

Yes. A teardown post on groas.com showed 309 claimed bot hits, but only six passed IP verification and just one was a live-answer fetch. Sorting a report by claimed hits alone would have given that post far more credit than the logs support.

### Does an AI bot fetching my page mean it was cited in an answer?

No. A verified live-answer fetch shows the page was pulled while a tool was assembling an answer, but it does not prove the page appeared in the finished answer or that anyone saw it. A successful page response only shows the page was available to read, so citations need to be checked separately through repeated prompt runs.

### Should I count all AI bot requests as one 'AI visits' number?

No. Training crawlers like GPTBot, search crawlers like OAI-SearchBot and user-triggered fetchers like ChatGPT-User do different jobs, and adding them together erases those distinctions. A verified training crawl means something different from a request made while a tool is assembling a live answer, so report the page and the verified fetcher together.

### Why isn't a prompt rank chart enough for AI search reporting?

A prompt tracker can show a client ranked third for a set of prompts, but it cannot show whether a verified fetcher pulled the client's page during the reporting period. Single prompt runs are also unstable, so the more defensible report pairs verified live-answer fetches by page and fetcher with citation frequency across repeated prompt runs.

## Paid Ads & SEO, Run For You

Businesses pay one flat monthly fee per service, agencies resell it under their own brand, and groas does the work either way.

[See Pricing](https://groas.com/pricing)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is a done-for-you marketing service, not software: elite people working with proprietary technology systems run search ads, Meta ads, SEO and AI search for businesses, and white label for agencies, with a dedicated senior strategist owning every account.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#webpage","url":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5","name":"Our AI Bot Logs Had 14,742 robots.txt Hits. Only 3,894 Were Verified.","description":"A 30-day look at groas.com edge logs shows why claimed bot hits mislead, where verified live-answer fetches concentrated, and what agencies should report instead.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2bc7480c-4fac-4cfb-be36-1f12c3917884.png"},"breadcrumb":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#breadcrumb"},"dateModified":"2026-10-26T05:26:15.303Z","mainEntity":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Our AI Bot Logs Had 14,742 robots.txt Hits. Only 3,894 Were Verified.","item":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#article","url":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5","mainEntityOfPage":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Our AI Bot Logs Had 14,742 robots.txt Hits. Only 3,894 Were Verified.","description":"A 30-day look at groas.com edge logs shows why claimed bot hits mislead, where verified live-answer fetches concentrated, and what agencies should report instead.","datePublished":"2026-10-26T05:26:15.225Z","dateModified":"2026-10-26T05:26:15.303Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2bc7480c-4fac-4cfb-be36-1f12c3917884.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"verified, hits, claimed, logs, page, bot, fetches, requests","wordCount":2058,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-5#faq","mainEntity":[{"@type":"Question","name":"How much AI bot traffic in server logs is actually verifiable?","acceptedAnswer":{"@type":"Answer","text":"On groas.com, only 3,894 of 14,742 requests to robots.txt that claimed an AI-crawler identity passed an IP verification check over 30 days, a 26.4% share. That means 73.6% of the claimed requests did not come from a published operator IP range."}},{"@type":"Question","name":"How do you verify that a request really came from an AI bot like ChatGPT or Claude?","acceptedAnswer":{"@type":"Answer","text":"Check the request's source IP against the operator's published IP ranges. A user-agent string is only a claim made by the requester, so verification changes the question from what the request called itself to whether it came from the operator it named."}},{"@type":"Question","name":"Which pages actually get fetched for live AI answers?","acceptedAnswer":{"@type":"Answer","text":"On groas.com, verified live-answer fetches concentrated on the homepage and two timely Google Ads changelog posts, which drew 561 and 361 fetches respectively, or 922 between them. The homepage accounted for 795 live-answer fetches, including 751 from ChatGPT-User."}},{"@type":"Question","name":"Can a page with lots of claimed bot hits have almost no real AI activity?","acceptedAnswer":{"@type":"Answer","text":"Yes. A teardown post on groas.com showed 309 claimed bot hits, but only six passed IP verification and just one was a live-answer fetch. Sorting a report by claimed hits alone would have given that post far more credit than the logs support."}},{"@type":"Question","name":"Does an AI bot fetching my page mean it was cited in an answer?","acceptedAnswer":{"@type":"Answer","text":"No. A verified live-answer fetch shows the page was pulled while a tool was assembling an answer, but it does not prove the page appeared in the finished answer or that anyone saw it. A successful page response only shows the page was available to read, so citations need to be checked separately through repeated prompt runs."}},{"@type":"Question","name":"Should I count all AI bot requests as one 'AI visits' number?","acceptedAnswer":{"@type":"Answer","text":"No. Training crawlers like GPTBot, search crawlers like OAI-SearchBot and user-triggered fetchers like ChatGPT-User do different jobs, and adding them together erases those distinctions. A verified training crawl means something different from a request made while a tool is assembling a live answer, so report the page and the verified fetcher together."}},{"@type":"Question","name":"Why isn't a prompt rank chart enough for AI search reporting?","acceptedAnswer":{"@type":"Answer","text":"A prompt tracker can show a client ranked third for a set of prompts, but it cannot show whether a verified fetcher pulled the client's page during the reporting period. Single prompt runs are also unstable, so the more defensible report pairs verified live-answer fetches by page and fetcher with citation frequency across repeated prompt runs."}}]}]}
```