Only 25% of Our Claimed AI Bot Hits Verified. The Page-Level Data Told a Different Story

Only 3,872 of 15,510 requests to our robots.txt that claimed to be AI bots verified. That is about 25%. I pulled 30 days of our edge logs expecting to see which pages assistants were reading; first I had to find out which requests belonged to assistants at all.

A User-Agent can say GPTBot/1.4 because someone typed it into a script. It is a claim, not an ID. After I checked source IPs against operator-published ranges or with forward-confirmed reverse DNS, I separated live-answer fetches from background crawls. The apparent winners changed. Some of the loudest paths were permission files and assets; a dated article had a much smaller hit count but a far stronger verification rate. Raw crawler counts were mostly noise.

That does not mean every unverified request was malicious, or that every verified fetch became a citation. It means a hit count alone cannot answer the question people usually ask of it: did an assistant fetch a page it might use to answer someone?

Three counts that should never share one column

I use three tiers, in order:

The distinction matters before any page ranking begins. Verification guidance describes the IP and DNS checks and the separate OpenAI bot identities. A verified GPTBot request belongs to a background crawler; a verified ChatGPT-User request is a different kind of event. OAI-SearchBot needs its own label, too. Rolling them together gives a larger number and a worse explanation.

I used to tell clients to count the bots together. I was wrong. In independent production-site measurements, 49.5% of 16,548 claimed crawler requests verified, while 67.8% of claimed user-triggered fetches were spoofed. Those are not rates to apply to our domain. They are a warning about the method: even a User-Agent that sounds like a live visit needs checking.

For our analysis, live-answer fetch means a verified request from a fetcher associated with a user-triggered action. It does not mean I saw the prompt, the answer, a citation or a click. Keep that boundary in mind; it is the difference between measuring the edge and inventing the rest of the journey.

The paths that looked busiest were not the cleanest

Our own 30-day edge-log pull is the source for every count in this table. The verification rates describe these paths on this domain during this window. They are not benchmarks for other sites. Approximate entries reflect how the paths were grouped in the pull, rather than a single exact path total.

Assets includes favicon.ico, Apple-touch icons, and the JavaScript, CSS and image files requested alongside pages. The grouped figures are useful for spotting a pattern, not for calculating a precise rate across all assets.

Robots.txt was the loudest path because it is a permission check. The 15,510 claimed hits did not represent 15,510 reads of our content, and the 3,872 verified hits still say nothing about which page an assistant might use. Asset counts were even less helpful: hundreds of claimed bot requests per file, with only roughly 1–4% verified. If I ranked URLs by claimed hits, I would put permissions and icons ahead of the article people might actually want to read.

There is a plausible reason to distrust that asset traffic. In independent tests, only 0.2% of spoofed requests read robots.txt, and 97% of spoofed sources also probed .env and configuration paths. That does not identify the source of each unverified request in our logs. It does explain why a familiar crawler name attached to an asset or a probe is not persuasive evidence of AI interest. Verify first; rank pages second.

The homepage shows why verification alone is not the final ranking, either. It had 2,026 verified hits and 821 live-answer fetches. The dated Google Ads updates post had fewer live-answer fetches in absolute terms: 529. I would not hide that comparison. What stands out is its profile: 853 of 872 claimed hits verified, or 97.9%, and 529 fetches came from the live-answer group. On the same site in the same month, a specific article held up under verification where permission checks and assets largely did not.

That is a better editorial signal than the raw hit leaderboard. The homepage still matters; it led this table in live-answer fetches. The dated post tells me something different: a page built around a particular update can draw substantial verified, user-triggered fetching without relying on homepage traffic. The log cannot tell me whether its date caused those fetches. It can tell me the page was pulled.

ChatGPT and Claude took different routes to the page

The assistant split adds another reason not to collapse the data. In our logs, ChatGPT-User fetched content directly: 786 homepage fetches and 528 fetches of that dated post, with few permission checks first. Claude-User recorded 930 robots.txt checks. Those checks belong in the permission column, not in a count of content reads, even though the request came from a user-triggered fetcher.

The difference is consistent with the operator behavior described in the draft’s source material. OpenAI says that, because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. A separate observed Claude-User test pulled /robots.txt before page fetches. Our logs show different request paths; they do not establish that every ChatGPT or Claude fetch will follow those sequences.

The operational point is narrower and more useful. If I am checking whether content is accessible, I test the two paths separately. A robots.txt check is not a page read. A successful content request is not a citation. Putting either one in the wrong column makes an access problem look like an editorial problem, or vice versa.

A fetch is a lead for content work, not proof of a citation

The dated post named a Google Ads change, attached a date and retained the old thresholds after the change. Service pages barely registered for live answers in our 30-day pull. That contrast gives me a place to investigate: the post contains the sort of specific fact an assistant could use when answering what changed and when; the service pages do not offer the same dated detail.

It does not let me say which prompt triggered a fetch, or that the post was cited. I cannot infer from these logs that a dated page always beats a service page. I can say that, in this pull, the dated page had 529 live-answer fetches while the service pages barely registered. If I am choosing what to inspect and improve next, that is enough to put the post near the top of the queue.

The next check needs a different kind of evidence. Server logs give me URLs, times and responses. Prompt tests can show what an assistant says for fixed questions. The measurement approach described here uses those views together rather than asking logs to stand in for the answer itself. I can compare the questions I care about with the pages that get fetched, then inspect whether those pages answer the questions clearly. I cannot reverse-engineer a private prompt from a server request. Use the log to choose a page to examine, not to declare a citation won.

What verification can—and cannot—settle

Thirty days on one domain exposed a large gap between claimed and verified traffic. It did not produce an AI-search revenue forecast. I cannot connect a particular request to a displayed sentence, a click, a lead or closed-won revenue. The edge records the pull, not what happened after it.

Independent measurements show how severe the identity problem can be for user-triggered traffic. One origin-log study counted 11 genuine Claude-User requests out of 318 claimed and zero genuine Perplexity-User requests out of 309 claimed. A separate live-fire test reported ChatGPT-User answering 11 requests and refusing 51, while Perplexity-User answered none of 39. Different tests measure different things; neither supplies a conversion rate for our site. Both are reasons to report what a count actually represents.

Verification has limits, too. The OpenAI verification guidance notes that GPTBot has published IP ranges but no stable reverse-DNS suffix. The Anthropic guidance describes its published bot file and the limits of its DNS and IPv6 coverage. I refresh the published files before each run and label requests that do not match as unverified, not human. If a stack relies on IP-file matching rather than a forward-confirmed lookup, the report should say that plainly.

Status codes have a similar boundary. A 200 on a content URL says the server returned the page. A 404 on a configuration path says that request did not retrieve that file. Neither status tells me whether an assistant answered a person. I would rather publish a smaller count with those limits attached than a spectacular count that quietly treats every request as a read.

The 30-day check before buying an AI-visibility report

This first pass needs edge logs and the operators’ published IP files, not a new visibility platform. Work in order so that a later chart does not inherit an earlier mistake:

1. Collect claimed bot requests. Keep the User-Agent, source IP, path, timestamp and status for the 30-day window; the User-Agent starts the check but cannot finish it.
2. Verify before counting operators. Match IPs to current published ranges and confirm reverse DNS where supported; label anything unmatched unverified.
3. Separate live-answer fetchers from background crawlers. Keep ChatGPT-User and Claude-User apart from GPTBot, OAI-SearchBot and ClaudeBot; do not count a permission check as a content read.
4. Rank content paths by verified live-answer fetches. Keep claimed and verified totals beside the ranking so the difference stays visible.
5. Inspect statuses and paths. A successful content response and a failed configuration-file probe are different events, even if both carry the same claimed User-Agent.
6. Refresh the IP files and repeat the check. Stale ranges can mislabel later traffic; the cost is losing the distinction that made the first report useful.

Buy the split, not the hit count

The decision the numbers support is straightforward: do not pay for AI visibility measured only in hits. Ask a tool or agency for claimed hits, verified hits with the IP-file date, and verified live-answer fetches by path and assistant. If it cannot produce that split, its busiest URLs may be robots.txt and icons while the useful content disappears in the total.

Our pull leaves me with two honest signals, not one inflated score. The homepage led these paths with 821 live-answer fetches. The dated post drew 529 and verified at 853 of 872 claimed hits. Assets sat near zero for live answers. I use that view to decide which existing page to inspect and what kind of specific answer is worth publishing next. I do not use it to claim a citation I cannot see.

I run the same check inside groas, where SEO and AI-search work runs continuously rather than arriving as a monthly PDF. But I would do the 30-day pull before spending anything: verify the bot, separate content fetches from permission checks and background crawls, then look at the pages assistants actually requested. Let the unverified 75% stay unverified. It has no business choosing your next page.