--- title: "We Counted AI Bot Hits for 30 Days. Only 26% Verified on robots.txt" description: "In 30 days of groas.com edge logs, robots.txt drew 14,905 claimed AI-crawler hits but only 3,886 verified. Here’s why answer-time fetches by page tell a more useful story than raw bot counts." url: "https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/da59d007-d684-4a20-a7a3-e62e6ae1634f.png" published: "2026-10-25T05:31:41.438Z" modified: "2026-10-25T05:31:41.497Z" --- October 25, 2026 · 10 min read # We Counted AI Bot Hits for 30 Days. Only 26% Verified on robots.txt [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![Cartoon bouncer at a 'robots.txt' door checks a queue of visitors all tagged 'GPTBot'; most are obvious impostors, and only a plain courier is real.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/da59d007-d684-4a20-a7a3-e62e6ae1634f.png) In this article 1. [Three counts that should not share a chart](#three-counts-that-should-not-share-a-chart) 2. [The 30-day log: plenty of knocks, little content](#the-30-day-log-plenty-of-knocks-little-content) 3. [What this sample cannot settle](#what-this-sample-cannot-settle) 4. [Logs tell you what was fetched. Prompts tell you what appeared.](#logs-tell-you-what-was-fetched-prompts-tell-you-what-appeared) 5. [A small-business setup without another dashboard](#a-small-business-setup-without-another-dashboard) 6. [Put three numbers on the monthly report](#put-three-numbers-on-the-monthly-report) Only **26%** of the AI-crawler hits on our `robots.txt` verified over 30 days. The other 74% were traffic _claiming_ to be an AI crawler, not traffic I could confirm as one. I used to tell clients to watch bot traffic like the total meant something. I was wrong. Anyone can put `GPTBot` in a user-agent string, and [OpenAI’s tokens can sit inside a normal Chrome user-agent](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). In one demo log, an IP check caught 3 spoofed IPs among 15 hits. Count by token, then check the IP against the vendor’s published ranges, and a lot of the apparent traffic disappears. Even the verified traffic needs another split. A crawler collecting pages and a bot fetching one while a user waits for an answer are doing different jobs. On our site, the homepage and two dated Google Ads posts accounted for most of the live-answer fetches visible in the path data. The useful question was not how many bots knocked. It was **which verified bots fetched which pages at answer time**. ## Three counts that should not share a chart I separate AI traffic into three measures, in order of how much they tell me about a page’s chance of appearing in an answer: 1. **Claimed hits:** requests with an AI-bot token in the user-agent. Our edge log looked for `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `Claude-SearchBot` and `PerplexityBot`. This catches candidates, not confirmed crawlers. In a separate 18-day Cloudflare sample, [113 requests claiming PerplexityBot and 112 claiming Kimi-SearchBot failed verification](https://pournasserian.com/writing/ai-friendly-website-2026). 2. **Verified hits:** claimed requests whose IPs match the relevant published ranges. The method uses [OpenAI’s bot-specific JSON files, Anthropic’s bots.json and the Perplexity files](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). Those lists change: in early October 2026, the same account of the method recorded [234 prefixes for ChatGPT-User, versus 18 for GPTBot, 39 for OAI-SearchBot and 38 for Anthropic](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). Verification is only as good as the list you use. 3. **Verified answer-time fetches:** requests from bots pulling a page in response to a user. [OpenAI describes ChatGPT-User as user-initiated](https://developers.openai.com/api/docs/bots.md). A fetch is not proof of a citation, but it is a much closer signal than a training crawl or an icon request. The distinctions matter even within one company’s bots. [OAI-SearchBot concerns ChatGPT search eligibility; GPTBot serves training; ChatGPT-User responds to user requests](https://developers.openai.com/api/docs/bots.md). Adding them together produces a large number with no single meaning. **Keep the three counts separate before you draw a conclusion from any of them.** ## The 30-day log: plenty of knocks, little content Our `robots.txt` drew 14,905 raw hits carrying an AI token and 3,886 verified hits. That gives us the 26% verification rate in the opening. It is a rate for this file on this site during this month, not an industry benchmark. The other conspicuous paths were less useful still: icon files drew roughly 700 claimed hits each, with fewer than 20 verified for each file. An icon request is not a read of a useful article, however impressive it looks in a bot-traffic chart. | Path or pattern | Claimed AI hits | Verified hits | How I read it | | --------------------------------------------------- | -----------------: | -----------------: | ------------------------------------------------------------ | | `/robots.txt` | 14,905 | 3,886 | About one in four claimed hits verified. | | Icon files: `apple-touch-icon`, `favicon`, `mstile` | Roughly 700 each | Fewer than 20 each | Asset requests, not evidence that content reached an answer. | | `llms.txt` pattern | Near zero verified | Near zero | Not a dependable visibility signal in this read. | That last row has a useful outside comparison, not a second result from our site. In one 17-day edge sample, [no verified AI crawler fetched `llms.txt`](https://pournasserian.com/writing/ai-friendly-website-2026). The same account cites an [Ahrefs study of 137,210 sites in which 97% of `llms.txt` files received zero requests](https://pournasserian.com/writing/ai-friendly-website-2026). Neither figure tells you what happened on every path in our export. They do explain why I would not use `llms.txt` requests as my headline metric. The first decision is simple: **remove unverified requests and static assets before calling a rise in bot hits a rise in AI visibility.** ### Three pages carried the visible answer-time fetches Once I looked at live-answer traffic by page, the chart changed. The homepage and two Google Ads update posts dominated the fetches visible in the path data. These are fetch counts, not a count of citations or recommendations; the [full fetch breakdown](https://groas.com/post/stop-writing-faq-posts-to-get-cited-by-c) provides the page-level context. | Page | Live-answer fetches | What the page offered | | ------------------------------- | ------------------: | ------------------------------------------- | | Homepage `/` | 804 | A direct account of what groas is and does. | | `/post/google-ads-updates-2026` | 556 | Dated, specific Google Ads changes. | | Second Google Ads update post | 349 | More dated detail on the same subject. | I would not turn that order into a universal content rule. It tells me where answer-time bots went _on this site_. It also shows why a site-wide verified total is not enough. Our [hub-page check](https://groas.com/post/postmortem-bots-read-our-category-pages) found pages with hundreds of verified fetches that fed zero live answers. Verified crawling and answer-time use can diverge sharply. The bots differed, too. In this read, ChatGPT-User came in bursts, fetched a page and left, consistent with [OpenAI’s description of a user-initiated request](https://developers.openai.com/api/docs/bots.md). Claude traffic leaned more toward repeated crawling, including sitemap requests. Another operator saw [ClaudeBot fetch a sitemap 122 times in 18 days](https://pournasserian.com/writing/ai-friendly-website-2026), though that is a separate site and sample. **Count answer-time fetches by page and assistant, not as one blended bot habit.** ## What this sample cannot settle These are edge logs from groas.com over 30 days. We publish dated Google Ads material, so the prominence of two update posts makes sense here. A plumber, a store with 4,000 SKUs or a SaaS business with gated documentation could have a different set of leading pages. I am showing the skew I can see, not predicting theirs. The path list I worked from was also truncated. The `robots.txt` and icon totals above are usable; the long tail is not a complete inventory. I can name the pages that led the visible answer-time fetches. I cannot claim that every omitted path got zero, or confidently rank a page that was cut from the export. Finally, IP verification depends on current vendor lists. With [234 ChatGPT-User prefixes in the early-October-2026 list cited above](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs), a stale copy can label real traffic as fake. That is a maintenance problem, not a reason to go back to trusting user-agents. **Trust the measured distinction, not an invented precision about the missing tail.** The 26% rate describes claimed versus verified `robots.txt` hits. The page table shows where the visible answer-time activity concentrated. Neither one tells us, by itself, what an assistant said to a buyer. ## Logs tell you what was fetched. Prompts tell you what appeared. A prompt tracker asks assistants a fixed set of buyer questions and records the answers. A server log records requests to your site. Most teams want the first result but try to infer it entirely from the second. The common [visibility-score calculation](https://searchengineland.com/measure-brand-visibility-ai-search-464524) is answers mentioning your brand divided by total answers, multiplied by 100. Run buyer-intent prompts repeatedly across the assistants you care about, and record mentions and citations separately. An assistant can mention groas while citing a third-party page, or cite our page without recommending us. Those are different outcomes with different content implications. I learned the same lesson in PPC: clicks and qualified pipeline were never interchangeable. Reporting the first as though it proved the second made a tidy deck and a bad decision. **A high bot-hit count with no live-answer fetches is the AI-search version of that mistake.** The log tells me which page had a chance to inform an answer. The prompt record tells me whether the brand appeared and whether its page was cited. Neither alone closes the loop. There is also a limit to what ordinary analytics sees. In another operator’s 18-day Cloudflare sample, [15,431 page fetches and 432 Markdown copies were counted at the edge](https://pournasserian.com/writing/ai-friendly-website-2026). JavaScript analytics missed those crawlers because they did not run scripts. That is an argument for edge data, not for treating every edge request as a buyer-facing result. ### Google needs a different line in the report Do not expect a `Google-Extended` hit to complete this picture. [Google-Extended has no HTTP user-agent; it is a robots.txt control for Gemini training](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). There is no `Google-Extended` request to count in a server log. That specific absence says nothing about whether your brand appeared in a Gemini answer. For AI Overviews, the draft’s cited reporting account describes a different limitation: [Search Console’s Generative AI reporting provides impressions by page, country, device and date across blended surfaces, without separated clicks](https://pournasserian.com/writing/ai-friendly-website-2026). A [UK-first rollout beginning in June 2026](https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results) also left many small accounts seeing impressions by page and date but not clicks or queries. An impression is useful, but it does not replace checking the answer itself. So I would keep two measurement paths: verified logs and answer-time fetches where they are observable, then a recurring prompt record for ChatGPT, Perplexity, Gemini and AI Overviews. Add the available Search Console view for Google. **Do not sell a string match in a server log as a Gemini visibility score.** ## A small-business setup without another dashboard You can build the first pass with an edge log and a Sheet. The work is less glamorous than a visibility score, which is usually a good sign. 1. **Collect at the edge.** [Cloudflare AI Crawl Control is available on all plans, and its GraphQL API can provide 31 days of counts by user-agent, verified category, path and status](https://pournasserian.com/writing/ai-friendly-website-2026). On Free, you lose the Accept header and referrers, but retain the path, claimed bot and verification information needed for this first split. 2. **Check identity and keep the check current.** Match claimed bots against published IP ranges rather than trusting their strings. For enforcement, [pair user-agent and verified IP ranges in WAF rules, maintain `robots.txt` as crawlers change, and watch request frequency and 403/429 responses](https://www.amicited.com/faq/how-do-i-configure-robots-txt-ai-crawlers/). I put the vendor JSON check on a monthly calendar reminder. Otherwise, a real bot can become a ‘fake’ in my report simply because my list aged. 3. **Record buyer-facing answers weekly.** Keep 10 to 30 prompts your buyers actually ask. Run them across ChatGPT, Perplexity and AI Overviews; record a brand mention and a citation in separate columns. If manual runs become tedious, the [cited n8n approach reads prompts from a Sheet, calls ChatGPT, Gemini and Perplexity APIs, and writes answers back](https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results). Its example puts 25 prompts a week in the low single-digit dollars per month, with Perplexity Sonar at $5 to $14 per 1,000 requests as the main cost. That is an example budget, not a price I measured for this site. 4. **Join the records monthly.** Start with pages that earned verified answer-time fetches. Check whether your tracked prompts produced citations, mentions, both or neither. A fetched-but-uncited page is worth inspecting before you rewrite it; a citation without a brand mention is a different problem from a page no assistant fetched at all. That gives each tool one job. **Logs establish the observed fetch; prompts establish the observed answer.** Neither needs to pretend to be the other. ## Put three numbers on the monthly report I would stop leading with raw bot hits and report these instead: | Measure | What it answers | What our read showed | | ----------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Verified share of claimed hits | How much of the apparent crawler traffic survived an IP check? | 3,886 of 14,905 `robots.txt` hits, or about 26%. | | Verified answer-time fetches by page | Which pages were pulled while answers were being assembled? | 804 for the homepage; 556 and 349 for two dated posts in the visible path data. | | Buyer-prompt mentions and citations, separately | Did the brand appear, and was its page used as a source? | The logs alone cannot answer this. It needs the recurring prompt record. | I would run those measures for 90 days before moving a content budget on the strength of them. If a report still leads with raw crawler counts, cut that line item first. Keep the dated pages assistants actually pull at answer time; inspect fetched-but-uncited pages before deciding what to rewrite. With $20k a month on the line, I want to know what had a chance to reach buyers and what appeared in their answers, not what knocked on `robots.txt`. ## Frequently Asked Questions ### Why did only 26% of the AI crawler hits on robots.txt verify? Over 30 days, robots.txt on groas.com drew 14,905 claimed AI-crawler hits but only 3,886 verified against vendor IP ranges, which works out to about 26%. Anyone can put GPTBot in a user-agent string, and OpenAI's tokens can sit inside a normal Chrome user-agent, so many hits claiming to be AI crawlers were not confirmed as such. ### How should I separate AI bot traffic counts in my logs? Split AI traffic into three counts: claimed hits (AI-bot tokens in the user-agent), verified hits (claimed requests whose IPs match the vendor's published ranges), and verified answer-time fetches (bots pulling a page while a user waits for an answer). Keep the three separate before drawing a conclusion, because adding them together produces a large number with no single meaning. ### What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User? According to OpenAI, OAI-SearchBot concerns ChatGPT search eligibility, GPTBot serves training, and ChatGPT-User responds to user-initiated requests. Because the three bots do different jobs, their request counts should be tracked and reported separately rather than summed. ### Which pages got the most answer-time bot fetches on the site? In the visible path data, the homepage led with 804 live-answer fetches, followed by two dated Google Ads update posts with 556 and 349. These are fetch counts, not citations or recommendations, and the skew reflects a site that publishes dated Google Ads material, so other kinds of sites could show a different pattern. ### Are icon file and llms.txt requests a sign my content is reaching AI answers? No. Icon files such as apple-touch-icon and favicon drew roughly 700 claimed hits each but fewer than 20 verified each, and an icon request is not a read of a useful article. The llms.txt pattern saw near-zero verified requests, and outside studies found most llms.txt files receive no requests at all, so neither is a dependable visibility signal. ### Can server logs tell me if my brand appears in AI answers? Server logs only record requests to your site; they tell you which pages had a chance to inform an answer. To learn whether the brand appeared and whether its page was cited, run a fixed set of buyer-intent prompts repeatedly across assistants like ChatGPT, Perplexity and AI Overviews, recording mentions and citations in separate columns. ### Can I track Google-Extended traffic in my server logs? No. Google-Extended has no HTTP user-agent; it is a robots.txt control for Gemini training, so there is no Google-Extended request to count in a server log. For AI Overviews, Search Console's Generative AI reporting provides impressions by page, country, device and date across blended surfaces, but without separated clicks. ### What is a simple low-cost setup to measure AI visibility? Collect AI traffic at the edge with Cloudflare AI Crawl Control, match claimed bots against published IP ranges, and refresh the vendor JSON lists on a monthly reminder so a stale list does not mislabel real bots. Then run 10 to 30 buyer prompts weekly across ChatGPT, Perplexity and AI Overviews, recording mentions and citations separately, and join the two records monthly. ## Paid Ads & SEO, Run For You Businesses pay one flat monthly fee per service, agencies resell it under their own brand, and groas does the work either way. [See Pricing](https://groas.com/pricing) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is a done-for-you marketing service, not software: elite people working with proprietary technology systems run search ads, Meta ads, SEO and AI search for businesses, and white label for agencies, with a dedicated senior strategist owning every account.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#webpage","url":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f","name":"We Counted AI Bot Hits for 30 Days. Only 26% Verified on robots.txt","description":"In 30 days of groas.com edge logs, robots.txt drew 14,905 claimed AI-crawler hits but only 3,886 verified. Here’s why answer-time fetches by page tell a more useful story than raw bot counts.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/da59d007-d684-4a20-a7a3-e62e6ae1634f.png"},"breadcrumb":{"@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#breadcrumb"},"dateModified":"2026-10-25T05:31:41.497Z","mainEntity":{"@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"We Counted AI Bot Hits for 30 Days. Only 26% Verified on robots.txt","item":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#article","url":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f","mainEntityOfPage":{"@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"We Counted AI Bot Hits for 30 Days. Only 26% Verified on robots.txt","description":"In 30 days of groas.com edge logs, robots.txt drew 14,905 claimed AI-crawler hits but only 3,886 verified. Here’s why answer-time fetches by page tell a more useful story than raw bot counts.","datePublished":"2026-10-25T05:31:41.438Z","dateModified":"2026-10-25T05:31:41.497Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/da59d007-d684-4a20-a7a3-e62e6ae1634f.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"verified, robots txt, hits, txt, page, robots, bot, days","wordCount":2019,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/we-logged-every-ai-bot-hit-on-our-site-f#faq","mainEntity":[{"@type":"Question","name":"Why did only 26% of the AI crawler hits on robots.txt verify?","acceptedAnswer":{"@type":"Answer","text":"Over 30 days, robots.txt on groas.com drew 14,905 claimed AI-crawler hits but only 3,886 verified against vendor IP ranges, which works out to about 26%. Anyone can put GPTBot in a user-agent string, and OpenAI's tokens can sit inside a normal Chrome user-agent, so many hits claiming to be AI crawlers were not confirmed as such."}},{"@type":"Question","name":"How should I separate AI bot traffic counts in my logs?","acceptedAnswer":{"@type":"Answer","text":"Split AI traffic into three counts: claimed hits (AI-bot tokens in the user-agent), verified hits (claimed requests whose IPs match the vendor's published ranges), and verified answer-time fetches (bots pulling a page while a user waits for an answer). Keep the three separate before drawing a conclusion, because adding them together produces a large number with no single meaning."}},{"@type":"Question","name":"What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?","acceptedAnswer":{"@type":"Answer","text":"According to OpenAI, OAI-SearchBot concerns ChatGPT search eligibility, GPTBot serves training, and ChatGPT-User responds to user-initiated requests. Because the three bots do different jobs, their request counts should be tracked and reported separately rather than summed."}},{"@type":"Question","name":"Which pages got the most answer-time bot fetches on the site?","acceptedAnswer":{"@type":"Answer","text":"In the visible path data, the homepage led with 804 live-answer fetches, followed by two dated Google Ads update posts with 556 and 349. These are fetch counts, not citations or recommendations, and the skew reflects a site that publishes dated Google Ads material, so other kinds of sites could show a different pattern."}},{"@type":"Question","name":"Are icon file and llms.txt requests a sign my content is reaching AI answers?","acceptedAnswer":{"@type":"Answer","text":"No. Icon files such as apple-touch-icon and favicon drew roughly 700 claimed hits each but fewer than 20 verified each, and an icon request is not a read of a useful article. The llms.txt pattern saw near-zero verified requests, and outside studies found most llms.txt files receive no requests at all, so neither is a dependable visibility signal."}},{"@type":"Question","name":"Can server logs tell me if my brand appears in AI answers?","acceptedAnswer":{"@type":"Answer","text":"Server logs only record requests to your site; they tell you which pages had a chance to inform an answer. To learn whether the brand appeared and whether its page was cited, run a fixed set of buyer-intent prompts repeatedly across assistants like ChatGPT, Perplexity and AI Overviews, recording mentions and citations in separate columns."}},{"@type":"Question","name":"Can I track Google-Extended traffic in my server logs?","acceptedAnswer":{"@type":"Answer","text":"No. Google-Extended has no HTTP user-agent; it is a robots.txt control for Gemini training, so there is no Google-Extended request to count in a server log. For AI Overviews, Search Console's Generative AI reporting provides impressions by page, country, device and date across blended surfaces, but without separated clicks."}},{"@type":"Question","name":"What is a simple low-cost setup to measure AI visibility?","acceptedAnswer":{"@type":"Answer","text":"Collect AI traffic at the edge with Cloudflare AI Crawl Control, match claimed bots against published IP ranges, and refresh the vendor JSON lists on a monthly reminder so a stale list does not mislabel real bots. Then run 10 to 30 buyer prompts weekly across ChatGPT, Perplexity and AI Overviews, recording mentions and citations separately, and join the two records monthly."}}]}]} ```