812 AI Fetches, Zero Live Retrievals: What Our Edge Logs Say About AI Visibility

Zero out of 812

None of the 812 verified AI requests to our category pages came from a user-directed fetcher. Crawlers visited those pages all month. When an assistant fetched a page in response to a live request, it went elsewhere. That gap is more useful to me than a composite AI visibility score.

Across 30 days of groas edge logs, user-directed agents accounted for 62.4% of verified AI requests to one technical post, 40.9% to the homepage, about 2% to the blog index, and 0% to category pages. An off-the-shelf score could give all four templates the same reassuring label: visible. The requests tell a less comfortable story.

We call the measure the live-answer ratio: verified requests from user-directed AI agents divided by all verified AI requests to a URL. The name needs a warning label. An edge log can show that a user-directed agent fetched a page. It cannot show that the assistant quoted the page, included it in its final answer, or served a buyer rather than another kind of user. I would rather work with that bounded signal than pretend a dashboard’s mention count reveals what happened on our site.

The 30-day view, before the explanation

Here are the four page types that made the difference plain. The source is our CDN edge request logs for 30 consecutive days, filtered for verified AI-agent requests. These are counts of requests, not unique people, prompts, citations, or sales.

One more count sat outside that table: Anthropic’s Claude-User agent requested /robots.txt 1,231 times. Those were permission-file requests, not 1,231 reads of a commercial page. Keeping them separate is the point.

This is an internal dataset, not a published cross-site benchmark. The logs support a comparison between these URLs over this period. They do not establish a typical ratio for other businesses, tell us which prompts caused individual requests, or measure how often an answer cited groas. Use the ratio to decide which of your pages warrant inspection, not to claim a share of AI answers.

First, separate a real agent from a convincing header

Most marketing analytics setups are a poor starting point for this analysis. They may miss traffic handled at the CDN edge, and a bot request is not necessarily recorded as a visit in the same way as a browser session. We pulled raw edge requests, then set aside traffic that failed our checks against provider IP information and reverse DNS. A user-agent string alone proves very little; anyone can put ChatGPT-User in a request header.

That caution is not theoretical. Centinel Analytica’s bot study reports that approximately 10% of HTTP requests claiming the ChatGPT-User string failed its verification checks. Its figure describes the traffic it examined, not ours. It explains why we filtered before counting rather than treating every impressive-looking header as an OpenAI visit.

We then grouped verified requests by what the agent was doing: training crawlers such as GPTBot and ClaudeBot, search indexers such as OAI-SearchBot, and user-directed fetchers such as ChatGPT-User and Claude-User. Swetrix’s explanation of AI crawler logs draws the same practical distinction between general crawling and user-triggered agents.

The categories matter more than the total. AI crawler statistics cited by Cloudflare Radar put bulk model-training collection at 47.62% of crawler requests, search-index updates at 8.26%, and live user-action fetchers at 3.66% in the traffic described there. Those percentages are not a breakdown of our logs; they show why a single bucket labelled AI bot hits can hide very different activity. Verify the requester, then classify the request.

The robots.txt heartbeat is not an answer count

Claude-User made 1,231 verified requests to our /robots.txt file during the window. Search Engine Journal’s account of Anthropic’s crawler structure explains why distinguishing that user-directed agent from Anthropic’s other crawlers matters for robots.txt rules.

A permission check tells us the agent checked whether it could access content. It does not tell us which substantive page it fetched next, whether it used that page in an answer, or whether each check represented a different person. I keep the 1,231 as an operational heartbeat. Folding it into content-fetch totals would inflate the visibility number while teaching us nothing about which page answered a question.

For a business asking how to measure brand visibility across ChatGPT, Gemini and Google AI Overviews, that distinction is worth more than another decimal place. Start with the requests that reached a content URL. Count permission checks separately.

The homepage drew live retrievals; the index mostly drew crawlers

Our homepage received 1,916 verified AI requests. Of those, 784 came from user-directed fetchers, predominantly ChatGPT-User. That makes its live-answer ratio 40.9%. The remaining 1,132 verified requests came from other AI-agent categories. Unveilr AI’s account of OpenAI fetch agents distinguishes GPTBot’s background collection from ChatGPT-User’s prompt-related retrieval, which is why I do not treat the two as interchangeable homepage attention.

A user-directed homepage fetch could follow a question about what groas does, what it charges, or how it compares with an agency. Those are plausible examples, not prompts recovered from our logs. The logs show the request and its target. They do not show the conversation or prove commercial intent. Still, 784 is a better starting point for inspecting homepage content than saying the page got nearly 2,000 AI visits and stopping there.

The blog index gives that comparison teeth. It received 1,048 verified AI requests, but just 21 user-directed fetches: roughly 2.0%. Its links may help crawlers discover posts. In this window, though, the index itself was rarely the page a user-directed agent requested. A useful navigation page need not be a useful answer page.

One technical post earned the strongest signal

The highest ratio among these page types belonged to a detailed, changelog-style product-architecture post. It recorded 697 verified AI requests, including 435 from user-directed agents: 62.4%. The page contains specific releases, operational changes, and explanations of how our models interact with campaign APIs. That gives a fetcher more to work with than a directory of links or a stack of promotional adjectives.

I can explain a plausible mechanism without pretending to know the prompt behind each hit. If someone asks whether groas executes campaign changes autonomously or merely displays recommendations, a page describing the mechanics is a more direct retrieval target than a blog index. Christian Rieber’s breakdown of AI crawler architectures describes the distinction between pre-indexed information and external retrieval when a system needs to ground an answer. Our logs show which of our URLs those user-directed agents fetched, not what the systems subsequently wrote.

That limit does not erase the finding. On this site, during this month, the specific technical page drew a much larger share of user-directed requests than the navigation templates. If I had one editorial hour to spend investigating that difference, I would inspect the pages that can answer a concrete question before polishing another archive introduction.

Zero on category pages is a priority signal, not a universal rule

The category pages received 812 verified AI requests and no user-directed fetches. Every verified request in this set fell outside the user-directed group. The blog index’s 21 of 1,048 looks busy beside that zero, but its roughly 2% ratio tells a similar story: most requests were not live retrievals.

I used to treat a crawl-heavy template as a straightforward editorial opportunity. More bot visits meant more reason to tune the page. This split makes that rule too crude. A category page can still help people navigate and help crawlers find articles. I would not delete one because its live-answer ratio is zero. I would stop spending editorial time polishing it for AI answers while a page containing the actual explanation needs work.

Nor can I promise that restructuring a category page will make an assistant fetch it. These logs contain no before-and-after test. They show where user-directed retrieval did and did not happen, which is enough to challenge an agency report that presents every bot hit as equivalent visibility.

A synthetic mention tracker answers a different question. It can run prompts and record whether a brand appears in the resulting output; it cannot use that result to tell you which real requests reached your server. Practitioners discussing commercial AI tracking tools on r/aeo also point to score volatility between runs. If you are choosing software to automate AI search visibility reporting, ask what it measures before treating its percentage as evidence of live retrieval.

Checklist: calculate the ratio without counting ghosts

For a team with CDN access, the calculation is simple; getting the inputs clean is the work. You do not need to turn this into an enterprise dashboard project. You do need to keep the distinctions above intact, especially if you are assessing AI search monitoring in-house.

1. Pull CDN edge request logs for a consistent period; origin logs can miss requests handled at the edge.
2. Separate /robots.txt checks from content-page requests; permission checks are not page views.
3. Group verified AI agents by training, indexing, and user-directed function.
4. Aggregate requests by canonical URL path and page template.
5. Divide verified user-directed fetches by all verified AI requests to each URL.
6. Review high-crawl, low-ratio pages for whether they contain an answer or mainly links; a low ratio alone does not prove a rewrite will help.
7. Track the same calculation by agent family over time; do not compare unlike buckets.
8. Verify claimed agents against provider IP information before trusting their user-agent headers. Skip this and spoofed requests can send your editorial work toward traffic that was never an AI assistant.

Spend on answers, not a prettier score

The numbers support a narrower, more useful decision than optimize everything for AI. Our technical post drew 435 user-directed fetches out of 697 verified AI requests; our category pages drew none out of 812. I would put the next round of content work into pages that can answer a specific technical or commercial question, while keeping navigation pages useful for navigation. Then I would watch the same verified ratio over time. I would not call a rising ratio a citation, a lead, or revenue without evidence for those outcomes.

That is where reporting-only work starts to look expensive. A score can alert you to a possible gap, but it cannot fix thin explanations, make a claim clearer, or coordinate the paid-search work that follows when organic visibility falls short. A polished dashboard is still a dashboard. I spent enough years making reports to know how easily the work after the report becomes somebody else’s six-month roadmap.

We built groas for that execution gap rather than as another layer of recommendations to interpret. Its specialized models work across paid and organic search within defined guardrails, while a named human strategist owns direction and accountability. The service runs on a flat monthly fee, with no setup fees and month-to-month terms. Those terms matter less than the operating question: does the work turn search spend and visibility into attributable pipeline and revenue?

For this dataset, the immediate call is plain. Do not buy a higher AI visibility score as a substitute for finding out which pages user-directed agents actually fetch. Check the edge logs, respect what they cannot prove, and put the work where the retrievals—and the unanswered business questions—are.