We Counted AI Bot Hits for 30 Days. Only 26% Verified on robots.txt

Only 26% of the AI-crawler hits on our robots.txt verified over 30 days. The other 74% were traffic claiming to be an AI crawler, not traffic I could confirm as one.

I used to tell clients to watch bot traffic like the total meant something. I was wrong. Anyone can put GPTBot in a user-agent string, and OpenAI’s tokens can sit inside a normal Chrome user-agent. In one demo log, an IP check caught 3 spoofed IPs among 15 hits. Count by token, then check the IP against the vendor’s published ranges, and a lot of the apparent traffic disappears.

Even the verified traffic needs another split. A crawler collecting pages and a bot fetching one while a user waits for an answer are doing different jobs. On our site, the homepage and two dated Google Ads posts accounted for most of the live-answer fetches visible in the path data. The useful question was not how many bots knocked. It was which verified bots fetched which pages at answer time.

Three counts that should not share a chart

I separate AI traffic into three measures, in order of how much they tell me about a page’s chance of appearing in an answer:

1. Claimed hits: requests with an AI-bot token in the user-agent. Our edge log looked for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot and PerplexityBot. This catches candidates, not confirmed crawlers. In a separate 18-day Cloudflare sample, 113 requests claiming PerplexityBot and 112 claiming Kimi-SearchBot failed verification.
2. Verified hits: claimed requests whose IPs match the relevant published ranges. The method uses OpenAI’s bot-specific JSON files, Anthropic’s bots.json and the Perplexity files. Those lists change: in early October 2026, the same account of the method recorded 234 prefixes for ChatGPT-User, versus 18 for GPTBot, 39 for OAI-SearchBot and 38 for Anthropic. Verification is only as good as the list you use.
3. Verified answer-time fetches: requests from bots pulling a page in response to a user. OpenAI describes ChatGPT-User as user-initiated. A fetch is not proof of a citation, but it is a much closer signal than a training crawl or an icon request.

The distinctions matter even within one company’s bots. OAI-SearchBot concerns ChatGPT search eligibility; GPTBot serves training; ChatGPT-User responds to user requests. Adding them together produces a large number with no single meaning. Keep the three counts separate before you draw a conclusion from any of them.

The 30-day log: plenty of knocks, little content

Our robots.txt drew 14,905 raw hits carrying an AI token and 3,886 verified hits. That gives us the 26% verification rate in the opening. It is a rate for this file on this site during this month, not an industry benchmark. The other conspicuous paths were less useful still: icon files drew roughly 700 claimed hits each, with fewer than 20 verified for each file. An icon request is not a read of a useful article, however impressive it looks in a bot-traffic chart.

That last row has a useful outside comparison, not a second result from our site. In one 17-day edge sample, no verified AI crawler fetched llms.txt. The same account cites an Ahrefs study of 137,210 sites in which 97% of llms.txt files received zero requests. Neither figure tells you what happened on every path in our export. They do explain why I would not use llms.txt requests as my headline metric.

The first decision is simple: remove unverified requests and static assets before calling a rise in bot hits a rise in AI visibility.

Three pages carried the visible answer-time fetches

Once I looked at live-answer traffic by page, the chart changed. The homepage and two Google Ads update posts dominated the fetches visible in the path data. These are fetch counts, not a count of citations or recommendations; the full fetch breakdown provides the page-level context.

I would not turn that order into a universal content rule. It tells me where answer-time bots went on this site. It also shows why a site-wide verified total is not enough. Our hub-page check found pages with hundreds of verified fetches that fed zero live answers. Verified crawling and answer-time use can diverge sharply.

The bots differed, too. In this read, ChatGPT-User came in bursts, fetched a page and left, consistent with OpenAI’s description of a user-initiated request. Claude traffic leaned more toward repeated crawling, including sitemap requests. Another operator saw ClaudeBot fetch a sitemap 122 times in 18 days, though that is a separate site and sample. Count answer-time fetches by page and assistant, not as one blended bot habit.

What this sample cannot settle

These are edge logs from groas.com over 30 days. We publish dated Google Ads material, so the prominence of two update posts makes sense here. A plumber, a store with 4,000 SKUs or a SaaS business with gated documentation could have a different set of leading pages. I am showing the skew I can see, not predicting theirs.

The path list I worked from was also truncated. The robots.txt and icon totals above are usable; the long tail is not a complete inventory. I can name the pages that led the visible answer-time fetches. I cannot claim that every omitted path got zero, or confidently rank a page that was cut from the export.

Finally, IP verification depends on current vendor lists. With 234 ChatGPT-User prefixes in the early-October-2026 list cited above, a stale copy can label real traffic as fake. That is a maintenance problem, not a reason to go back to trusting user-agents.

Trust the measured distinction, not an invented precision about the missing tail. The 26% rate describes claimed versus verified robots.txt hits. The page table shows where the visible answer-time activity concentrated. Neither one tells us, by itself, what an assistant said to a buyer.

Logs tell you what was fetched. Prompts tell you what appeared.

A prompt tracker asks assistants a fixed set of buyer questions and records the answers. A server log records requests to your site. Most teams want the first result but try to infer it entirely from the second.

The common visibility-score calculation is answers mentioning your brand divided by total answers, multiplied by 100. Run buyer-intent prompts repeatedly across the assistants you care about, and record mentions and citations separately. An assistant can mention groas while citing a third-party page, or cite our page without recommending us. Those are different outcomes with different content implications.

I learned the same lesson in PPC: clicks and qualified pipeline were never interchangeable. Reporting the first as though it proved the second made a tidy deck and a bad decision. A high bot-hit count with no live-answer fetches is the AI-search version of that mistake. The log tells me which page had a chance to inform an answer. The prompt record tells me whether the brand appeared and whether its page was cited. Neither alone closes the loop.

There is also a limit to what ordinary analytics sees. In another operator’s 18-day Cloudflare sample, 15,431 page fetches and 432 Markdown copies were counted at the edge. JavaScript analytics missed those crawlers because they did not run scripts. That is an argument for edge data, not for treating every edge request as a buyer-facing result.

Google needs a different line in the report

Do not expect a Google-Extended hit to complete this picture. Google-Extended has no HTTP user-agent; it is a robots.txt control for Gemini training. There is no Google-Extended request to count in a server log. That specific absence says nothing about whether your brand appeared in a Gemini answer.

For AI Overviews, the draft’s cited reporting account describes a different limitation: Search Console’s Generative AI reporting provides impressions by page, country, device and date across blended surfaces, without separated clicks. A UK-first rollout beginning in June 2026 also left many small accounts seeing impressions by page and date but not clicks or queries. An impression is useful, but it does not replace checking the answer itself.

So I would keep two measurement paths: verified logs and answer-time fetches where they are observable, then a recurring prompt record for ChatGPT, Perplexity, Gemini and AI Overviews. Add the available Search Console view for Google. Do not sell a string match in a server log as a Gemini visibility score.

A small-business setup without another dashboard

You can build the first pass with an edge log and a Sheet. The work is less glamorous than a visibility score, which is usually a good sign.

1. Collect at the edge. Cloudflare AI Crawl Control is available on all plans, and its GraphQL API can provide 31 days of counts by user-agent, verified category, path and status. On Free, you lose the Accept header and referrers, but retain the path, claimed bot and verification information needed for this first split.
2. Check identity and keep the check current. Match claimed bots against published IP ranges rather than trusting their strings. For enforcement, pair user-agent and verified IP ranges in WAF rules, maintain robots.txt as crawlers change, and watch request frequency and 403/429 responses. I put the vendor JSON check on a monthly calendar reminder. Otherwise, a real bot can become a ‘fake’ in my report simply because my list aged.
3. Record buyer-facing answers weekly. Keep 10 to 30 prompts your buyers actually ask. Run them across ChatGPT, Perplexity and AI Overviews; record a brand mention and a citation in separate columns. If manual runs become tedious, the cited n8n approach reads prompts from a Sheet, calls ChatGPT, Gemini and Perplexity APIs, and writes answers back. Its example puts 25 prompts a week in the low single-digit dollars per month, with Perplexity Sonar at $5 to $14 per 1,000 requests as the main cost. That is an example budget, not a price I measured for this site.
4. Join the records monthly. Start with pages that earned verified answer-time fetches. Check whether your tracked prompts produced citations, mentions, both or neither. A fetched-but-uncited page is worth inspecting before you rewrite it; a citation without a brand mention is a different problem from a page no assistant fetched at all.

That gives each tool one job. Logs establish the observed fetch; prompts establish the observed answer. Neither needs to pretend to be the other.

Put three numbers on the monthly report

I would stop leading with raw bot hits and report these instead:

I would run those measures for 90 days before moving a content budget on the strength of them. If a report still leads with raw crawler counts, cut that line item first. Keep the dated pages assistants actually pull at answer time; inspect fetched-but-uncited pages before deciding what to rewrite. With $20k a month on the line, I want to know what had a chance to reach buyers and what appeared in their answers, not what knocked on robots.txt.