Only 26% of Our Claimed AI Bot Fetches Were Verified

Only 26% of the AI crawler fetches we counted passed verification. Our edge logs showed 14,905 robots.txt requests claiming AI crawler identities in 30 days; 3,886 matched both the claimed user-agent and the operator’s IP range. Put the first number in a client report without the second, and you are selling impressions again.

I used to tell clients a bot hit was a bot hit. I was wrong. A user-agent string is a claim, not an ID card: verification needs the user-agent and IP to match. Browser analytics do not fill the gap, either. In one large crawler-tracking dataset, more than 500 million GPTBot fetches executed no JavaScript. The result is an awkward pair of blind spots: a string-based tool can count requests it cannot authenticate, while a JavaScript tag can miss crawler requests entirely.

The 14,905-to-3,886 gap changes what a hit means

Most AEO dashboards offer a bot total and call it visibility. I need three layers before that total tells me much. Claimed hits identify themselves with an AI-related user-agent. Verified hits also match an operator IP range. Live-agent fetches come from agents involved in user-facing retrieval. Those layers describe different events, not three names for the same success.

The first two rows below come from our 30-day edge-log review. The last row counts requests to robots.txt, not successful article retrievals or citations.

The remaining claimed requests did not pass verification. That does not prove every one was malicious or spoofed; it means I would not present them as authenticated bot activity. Spoofing is easy enough to make the distinction necessary. A user-agent takes one header edit to imitate, whereas the operator’s IP range provides the stronger identity check. In another monitored dataset, 139,943 requests claimed a trusted crawler identity and failed verification.

One post in our logs makes the reporting problem less abstract: 344 claimed AI bot hits and zero verified hits. A string-only dashboard makes that post look busy. Our verification found no confirmed crawler among those hits. Report the verified number beside the claimed number, or leave the claimed number out of the client-facing headline.

Live fetches favored posts, not category pages

A verified bulk crawl and a live-answer fetch are not interchangeable. GPTBot, OAI-SearchBot and ChatGPT-User have distinct jobs: bulk crawling, search indexing and fetching a page in response to a person’s request. I would not treat an OAI-SearchBot index fetch as proof that a particular answer used the page. Even a ChatGPT-User fetch shows retrieval activity, not what appeared in the final answer.

That limit matters when reading our page-level results. In the same 30-day edge-log review, live-agent activity concentrated on the homepage and individual posts. Category pages had no live-answer fetches in this sample.

Same site, same month, different treatment. I wrote up the category-page result in Postmortem: AI Bots Fetched Our Hub Pages Hundreds of Times. They Fed Zero Live Answers. Bulk crawlers can work through a hub; the live fetches we observed went to pages that stood on their own. That does not prove every standalone post will earn a citation. It does give an agency a better content question than “How many pages did bots crawl?”: Which individual pages did live agents actually fetch?

Other logs show why I keep fetches separate from outcomes. Reflexa recorded 115 live retrievals but only two AI-referred visits in analytics. That is not a universal visit rate, and it does not reveal how many answers cited the site. It shows why a visit report cannot stand in for a retrieval report. Use page-level fetches to decide what to inspect; use answer monitoring to find out what the model said.

A robots.txt check is not an article fetch

Our robots.txt received 1,241 live-agent requests during the window. It is tempting to label those “answer opportunities.” I would label them permission checks. An agent can check robots.txt and then fail to fetch the intended page because of a block, rate limit or redirect. The check tells you where to look next, not that content reached an answer.

Access failures can be large enough to make content work beside the point. In one seven-day Cloudflare sample, 29,099 bot requests included 65.8% attributed to AI bots. Its reported rate-limit figures included 29% for ClaudeBot, 29% for GPTBot, 51% for Amazonbot and 61% for Bytespider. On that site, the author also reported 37.8% citation presence in Google AI Mode, where access was 100%, versus 0% in Claude when cache-miss fetches were blocked. Those are observations from one site, not a promised result of changing a firewall rule. They are a reason to inspect access before commissioning a rewrite.

Nor should an agency assume every operator or agent meets the same rule. A 40-site access study found that 18% blocked GPTBot and 12% blocked OAI-SearchBot. Training crawlers and user-facing agents can face different permissions. A single “AI bots allowed” checkbox conceals that distinction.

For a retainer, the order is plain: check server responses for 403s, 429s and redirects before changing a word of the page. Access errors in logs can explain a citation drop when a dashboard cannot. I have seen agencies pitch new posts while existing posts were unreachable. Fix access first. Then argue about wording.

What belongs in the agency report

If I ran AEO for a client using these logs, I would not put a claimed-hit total in a large typeface and call it progress. The reporting table needs verified fetches and live-agent fetches by page, with access errors beside them. Answer appearances belong on a separate line because a crawler log records what was requested, not what a model ultimately said.

That separation matters commercially. Agencies already know how easy it is to foreground defensible vanity metrics: impressions, keyword totals and raw traffic look tidy in a PDF. Raw bot hits are the same habit with a new label. A claimed hit does not tell the client whether a buyer saw their name in an answer. A verified fetch does not tell them that either, but at least it establishes which crawler activity the agency can stand behind.

I would also keep prompt monitoring distinct from crawler measurement. API monitoring and checks of the consumer interface can differ by 5% to 15% for the same brand; a mention, a citation and a visibility score are not the same result. Ask a vendor what each metric counts and how it is calculated. Crawler logs tell you what an AI system could fetch; answer monitoring tells you what it said. Neither should borrow the other’s label.

For content planning, our sample pushes me toward a standalone answer to each buyer question rather than another category hub. The live fetches went to individual posts; the category pages drew no live use in our window. I would test that pattern on a client’s own pages before making it a universal rule. Build the answer on the page a live agent fetches, then measure whether that page appears in tracked answers.

Checklist: four questions before you resell an AEO platform

If a platform’s number goes in your client report, you own the explanation when the client asks what it bought them. I would run these checks before selling the dashboard. The first two protect the measurement; the last one protects your ability to deliver the work.

1. Does it measure at the edge or through a JavaScript tag? A tag will miss crawler fetches that do not execute JavaScript.
2. Does it separate verified bots from claimed identities? Ask for user-agent and IP verification, or our 344-claimed, zero-verified post becomes a reported win.
3. Does it break out live agents by page? Keep user-triggered retrieval separate from bulk crawls and index fetches; do not label any fetch a citation.
4. Can you onboard and act without a client-side engineering project? The check agencies skip most: proxy changes, ticket queues and CMS access can consume the margin before the first report. Look for a connection the client can complete, a multi-client workspace and branded reporting, then test whether the pricing works at scale. A 2026 agency-tool guide lists prices from $25 to $500 a month depending on seats and add-ons. Skip this check and you can end up selling reporting without a workable delivery process.

A dashboard also cannot promise that retrievals will become visits. The 115-to-two split in Reflexa’s logs is enough to make me wary of selling a traffic forecast from fetch counts. I would report which tracked prompts produced an answer appearance, with or without a link, alongside the verified and live-agent activity that preceded it. Otherwise the calendar can stay empty while the crawler chart rises.

Winning content needs upkeep, too. One prompt-monitoring guide puts a useful refresh window at roughly eight to 14 weeks; I would treat that as a review cue, not an expiry date. Recheck the pages live agents fetch, update stale details, tighten the answer and confirm the host still serves the page instead of a new 429. This is where groas earns its place for agencies: tracking without execution leaves the same recommendation waiting in the next report. Continuous execution, with actions logged under the agency’s brand, makes the measured work something the agency can maintain. Sell maintenance of fetchable, useful pages, not a one-time stack of hubs.

The decision the numbers support

After that month, I changed what goes in a client report and what stays out. Claimed hits stay out of the headline. For each important page, I want verified fetches, live-agent fetches and access errors. Below those sits the outcome check: did the brand appear in an answer to a tracked prompt, and was there a link? The log lines help investigate that result. They cannot substitute for observing it.

The order also stops two familiar mistakes. If verified fetches rise but citations do not, inspect the page and the answers before claiming success. If fetches fail because the host returns a 403 or 429, rewriting the opening paragraph will not fix access. And if claimed hits spike while verification stays flat, there is nothing to celebrate in the monthly PDF.

My rule for an agency retainer is therefore blunt. If a platform cannot measure at the edge, separate verified from claimed identities and show live-agent activity by page, do not put its bot total in front of a client. Our 14,905 claimed robots.txt requests yielded 3,886 verified ones. Our live-agent fetches favored standalone posts while category pages logged none. Those numbers support better reporting and a clearer content priority. They do not, on their own, prove a citation or a sale.

If you do one thing Monday, pull the last 30 days of your own edge logs and put claimed, verified and live-agent requests side by side. Then inspect the standalone page with the most live-agent fetches: can it be fetched, and does it answer the question without sending the reader through a hub? That is a more useful decision than buying another chart of impressions with a bot-shaped label.