---
title: "675 Live-Answer Fetches of robots.txt. Still Not a Recommendation."
description: "The groas snapshot records 675 live-answer fetches for robots.txt and 387 for an article. Here’s what those counts prove—and what requires saved answers."
url: "https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/bb2741c5-4a07-4dbd-b2e2-a1ebe0be6f10.png"
published: "2026-11-17T05:14:22.702Z"
modified: "2026-11-17T05:14:22.750Z"
---

November 17, 2026 · 9 min read

# 675 Live-Answer Fetches of robots.txt. Still Not a Recommendation.

[](https://groas.com/author/alexander-perelman)

[Alexander Perelman](https://groas.com/author/alexander-perelman)Personally Tested

Head Of Product @ groas

[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Cartoon shopkeeper proudly clicks a tally counter as visitors read his 'House Rules' sign, then walk across the street into a rival's shop.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/bb2741c5-4a07-4dbd-b2e2-a1ebe0be6f10.png)

In this article

1. [Start with the counters, not the biggest total](#start-with-the-counters-not-the-biggest-total)
2. [robots.txt beat the article. That is the warning.](#robotstxt-beat-the-article-that-is-the-warning)
3. [The article and RSS feed show why verification comes first](#the-article-and-rss-feed-show-why-verification-comes-first)
4. [The homepage’s 763 fetches still cannot answer “Were we recommended?”](#the-homepages-763-fetches-still-cannot-answer-were-we-recommended)
5. [An incomplete path list cannot establish a content gap](#an-incomplete-path-list-cannot-establish-a-content-gap)
6. [Citation rates need answers in the denominator](#citation-rates-need-answers-in-the-denominator)
7. [Buy evidence and execution, not a larger fetch counter](#buy-evidence-and-execution-not-a-larger-fetch-counter)

In the supplied groas snapshot, `/robots.txt` received 675 live-answer fetches in 30 days; a Google Ads updates article received 387. The crawling rulebook got more retrieval activity than the article, which is why I would not mistake a fetch count for a recommendation.

A business installs AI-visibility tracking, sees activity climb, and wants to know why ChatGPT, Gemini or Perplexity still does not recommend it. The confusion starts with what the report measures. Server logs record [which AI crawlers visited and what they fetched](https://optiseon.com/blog/ai-crawler-log-analysis/). They do not record what an assistant ultimately told the person asking the question.

**Even a verified fetch associated with a live answer proves retrieval, not citation or recommendation.** That distinction is the story in these numbers. Everything else depends on keeping it intact.

## Start with the counters, not the biggest total

The snapshot covers 30 days, but its displayed path list is incomplete. The figures below come from that supplied snapshot, not from the explanatory articles linked throughout this analysis. It contains retrieval counts, not saved assistant answers.

Before comparing pages, I separate the report’s counters:

| Counter          | What it measures                                            | What it does not prove                     |
| ---------------- | ----------------------------------------------------------- | ------------------------------------------ |
| `hits`           | A request arrived claiming a known AI user-agent            | That the bot was genuine                   |
| `verifiedHits`   | A request passed an IP-range or reverse-DNS check           | That the page contributed to an answer     |
| `liveAnswerHits` | A verified fetch was associated with live-answer generation | That the business was cited or recommended |

The identity check matters. A user-agent is a claim, not identification. Verification uses [IP ranges or reverse DNS rather than the user-agent string alone](https://optiseon.com/blog/ai-crawler-log-analysis/). A busy endpoint with almost no verified requests is not evidence of heavy assistant interest.

Nor are fetches unique users, unique answers or recommendations. An assistant can retrieve the same URL repeatedly while assembling an answer. Scheduled polling can also accumulate requests without a buyer behind each one.

I used to tell clients that crawl frequency meant interest. I was wrong. Crawl frequency means accessibility plus machine habit. **Verify the visitor first; then ask what kind of activity the count represents.**

## robots.txt beat the article. That is the warning.

Here are the relevant rows from the snapshot. A dash means the figure is not supplied here, not that the count is zero.

| Path or page               | Hits | Verified hits | Live-answer fetches |
| -------------------------- | ---: | ------------: | ------------------: |
| `/robots.txt`              |    — |             — |                 675 |
| Google Ads updates article |  665 |           652 |                 387 |
| Blog RSS endpoint          |  397 |             3 |                   0 |
| Homepage                   |    — |             — |                 763 |

The robots.txt result is plausible once the mechanism is clear. Access checks can accompany retrieval without becoming sources in the final answer. For example, [PerplexityBot follows robots.txt, while Perplexity-User generally ignores it because a person requested the page](https://www.ansvisor.com/ai-visibility-glossary/perplexitybot-perplexity-user). Different fetching agents do different jobs. Treating all their activity as buyer attention erases that distinction.

The snapshot establishes that the rulebook was fetched frequently during live-answer work. It does not establish that those requests produced recommendations. A file containing crawling instructions is not evidence that an assistant selected the business as the best vendor.

That is the useful comparison with the article. The article’s fetches concern substantive content; the robots.txt fetches concern access instructions. Both sit in the same live-answer column. Neither row contains the answer itself.

An assistant might retrieve an article, find a clearer source and leave it out. It might use the article for background while citing another page. As the crawler glossary puts it, [access gets you in the door and nothing more](https://www.ansvisor.com/ai-visibility-glossary/perplexitybot-perplexity-user).

**The live-answer label narrows the activity being measured. It does not close the gap between retrieval and use.** Keep the count, but do not promote it into an outcome.

## The article and RSS feed show why verification comes first

The updates article and RSS endpoint make a different distinction visible: claimed bot traffic versus verified retrieval.

| Measure                       | Google Ads updates article | Blog RSS endpoint |
| ----------------------------- | -------------------------: | ----------------: |
| Claimed AI requests           |                        665 |               397 |
| Verified requests             |                        652 |                 3 |
| Approximate verification rate |                        98% |          Under 1% |
| Recorded live-answer fetches  |                        387 |                 0 |

The percentages are simple divisions of the supplied counts: verified requests divided by hits. They describe verification within these rows, not citation performance.

Rank the rows by raw hits and RSS looks busy. Apply verification and almost all of its apparent activity disappears. The article, by contrast, retains nearly all its requests through the identity check and records repeated retrieval associated with live answers.

That does not tell me exactly what generated every unverified RSS request. It tells me not to count those requests as confirmed AI activity. There is no need to invent an explanation when the measurement already supplies the decision.

The article’s stronger evidence also has a ceiling. Its verified live-answer fetches establish retrieval during answer construction. They still do not establish inclusion in the finished response.

**Hits measure who knocked; verified hits measure who proved identity. Neither measures who recommended the business.** Any report that ranks visibility by the first column needs that distinction beside the chart, not buried underneath it.

## The homepage’s 763 fetches still cannot answer “Were we recommended?”

The homepage recorded more live-answer fetches than either robots.txt or the updates article. That establishes frequent retrieval while answers were being built. It does not establish what those answers said about groas.

I keep these outcomes separate:

- **Mention:** the answer names the business.
- **Citation:** the answer links to the site as a source.
- **Recommendation:** the answer presents the business as a choice worth considering.

A citation is not automatically a recommendation. A mention is not automatically favorable. A homepage fetch proves none of them on its own.

The companion breakdown of [905 live-answer fetches and how many became brand recommendations](https://groas.com/post/905-live-answer-fetches-are-not-905-brand-recommendations) keeps retrieval and recommendation in separate columns for this reason. One is a server-side observation. The other needs evidence from the answer.

This rollup has no prompt text, model version, geography, saved response or list of sources actually cited. It cannot connect a homepage request to a buyer-ready recommendation. Say the search budget is $20k a month: hundreds of fetches still cannot tell me whether the assistant recommended the business or read its pages on the way to recommending a competitor.

Monitoring tools need the same discipline. Their test prompts measure [potential visibility, not actual user exposure or traffic](https://blog.frizerly.com/26797/-best-chatgpt-monitoring-tools-for-lean-saas-marketing-teams). Saved test answers can establish what happened in those tests. They are not a census of what every buyer saw.

**Until the answer is available, call the homepage total retrieval—not visibility won.**

![Server log rows for robots.txt beside an empty AI answer with no citation](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/75fe53ea-a6f6-494b-bdb3-8e6f7bbd2b07.png)

To establish what happened in an observed answer, I ask for:

1. **The saved answer text**, so the wording can be inspected.
2. **The source URLs shown**, so a mention is not mistaken for a citation.
3. **The exact prompt and engine**, so the result has a defined context.
4. **The timestamp**, so the observation can be checked against later runs.

Model version and geography belong with that record when available. Without the answer itself, extra dashboard detail does not repair the missing evidence.

## An incomplete path list cannot establish a content gap

The snapshot’s path list is truncated. If a service, pricing or comparison page does not appear, the only supported conclusion is that it is absent from the displayed rows.

It might have no fetches. It might have activity below the display limit. The excerpt cannot decide between them.

That matters because a missing row can quickly become a content recommendation: “The assistant is ignoring this topic, so publish more.” The premise has not been established. Before commissioning content to fix a supposed retrieval gap, I want the complete export.

The companion method that [separates claimed AI-bot crawls from verified fetches](https://groas.com/post/545-ai-bot-requests-three-verified-what-white-label-seo) reinforces the same discipline: work from verified rows that are present, not assumptions about rows that are missing.

Even a complete retrieval export would answer an access-and-fetching question, not settle which topics win recommendations. That requires observed answers as well.

**Use the visible rows to choose what to investigate. Do not turn an incomplete list into a map of everything the assistant ignores.**

## Citation rates need answers in the denominator

The other tempting shortcut is to divide citations by fetches. I would reject that percentage before discussing whether it looks good.

Fetches count retrieval events. The same answer may involve repeated requests, and this rollup combines activity across unknown prompts, models and days. An answer-level outcome needs an answer-level denominator.

| Question                                              | Relevant measure                                | What the snapshot supplies                    |
| ----------------------------------------------------- | ----------------------------------------------- | --------------------------------------------- |
| Did verified AI agents fetch the page?                | Verified requests                               | Yes, for the displayed rows with those counts |
| Was retrieval associated with live-answer generation? | Live-answer fetches                             | Yes, as classified in the snapshot            |
| How often did checked answers cite the domain?        | Answers linking to the domain ÷ answers checked | Not supplied                                  |
| Did the assistant recommend the business?             | Saved answer text and its framing               | Not supplied                                  |

For citation monitoring, the relevant calculation is [answers linking to the domain divided by answers checked, tracked per model over time with fixed prompts](https://www.storkim.com/guides/ai/monitor-ai-visibility-chatgpt-gemini-perplexity/). Its limit matters too: that rate describes the answers checked under that monitoring setup. It does not measure every real-world answer.

Manual checks, analytics, logs and monitoring tools therefore answer different questions. Manual checks show a particular response. Analytics shows clicks. Logs show retrieval. Repeated prompt tracking shows citation patterns within the test set, because [a bot can read a page and never use it](https://www.bestgeosoftware.com/blog/how-do-i-find-out-if-ai-agents-are-citing-my-content/).

**A fetch total can prioritize pages to test. It cannot serve as the denominator of a citation rate.** A percentage with mismatched units is still bad measurement, however tidy the slide looks.

## Buy evidence and execution, not a larger fetch counter

If the only question is whether bots can reach the site, verified server logs may be enough. There is no reason to buy a broader service to answer a narrower question.

If the question is why the business does not appear in recommendations across ChatGPT, Gemini and Perplexity, retrieval logs leave the important work unfinished. I need observed answers, separated by engine and tracked over time. A fetch-heavy page does not tell me which competitor appeared instead, how the assistant framed the options or whether anyone clicked through.

My buying checklist is short:

- Saved answers with their displayed source links, not fetch totals presented as citations.
- Exact prompts, engines, timestamps and available model and geography details.
- Citation rates calculated from checked answers, separately by engine.
- Repeated appearances before declaring a source consistently adopted, rather than celebrating a single spike.
- Referral and enquiry reporting alongside visibility, so attention is not mistaken for business value.

Then comes execution. Technical access fixes help the right pages get fetched and parsed. Buyer-focused content addresses the question better than the page currently winning. Authority building addresses trust. None of those jobs is completed by producing another report that says the bot visited.

Most retainers still charge 2015 prices for checking this once a month. The deck calls it coverage. I call it two reports describing the same gap.

The groas alternative is [continuous execution across ads, SEO and AI search](https://groas.com/), with changes logged, a human owner accountable for the outcome, and a flat fee rather than a percentage of spend, as shown on the pricing page. That is the operating model I would buy: work on the problem, not another presentation of it.

The snapshot supports a clear decision. Keep the fetch log as evidence of retrieval and a way to prioritize investigation. **Pay for saved citations, repeated recommendations and attributable enquiries—not for relabeling a bot reading the rulebook as a customer choosing the business.**

## Frequently Asked Questions

### Does a live-answer fetch mean AI tools are recommending my business?

No. A live-answer fetch only proves that an AI crawler retrieved a page while an answer was being generated. It does not prove the page was cited, linked to, or recommended in the finished answer. Recommendation requires evidence from the answer itself, such as saved answer text and its displayed sources.

### What is the difference between hits, verified hits and live-answer fetches in AI crawler reports?

A hit is a request arriving with a claimed AI user-agent, which does not prove the bot is genuine. A verified hit passed an IP-range or reverse-DNS identity check. A live-answer fetch is a verified request associated with live-answer generation. None of the three proves that a page was cited in an answer or that the business was recommended.

### Why would robots.txt get more live-answer fetches than an actual article?

Because AI agents fetch the crawling rulebook as part of access checks that accompany retrieval, and different agents do different jobs. PerplexityBot follows robots.txt, for example, while Perplexity-User generally ignores it because a person requested the page. Those fetches concern access instructions, not content selection, so they say nothing about whether the business was recommended.

### Why should I verify AI bot requests before counting them?

Because a user-agent string is a claim, not identification. In the snapshot described, the Google Ads article kept roughly 98 percent of its claimed requests through verification, while the RSS endpoint kept under 1 percent. Unverified requests should not be counted as confirmed AI activity, so verification changes what the numbers actually mean.

### If my homepage has hundreds of AI fetches, does that mean ChatGPT recommends me?

No. The homepage can record the most live-answer fetches and still not establish what any answer said about the business. A mention, a citation and a recommendation are separate outcomes, and a homepage fetch proves none of them on its own. Until the saved answer is available, call the total retrieval rather than visibility won.

### A page is missing from my AI visibility report, so should I publish more content on that topic?

Not on that evidence alone. If a page does not appear in the report's path list, the only supported conclusion is that it is absent from the displayed rows. It might have no fetches or activity below the display limit. Request the complete export before commissioning content to fix a supposed retrieval gap.

### Can I calculate a citation rate by dividing citations by fetches?

No. Fetches count retrieval events, and the same answer may involve repeated requests across unknown prompts, models and days, so the units do not match. A proper citation rate is answers linking to the domain divided by answers checked, tracked per model over time with fixed prompts. Fetch totals are better used to prioritize which pages to test.

## Run Your Marketing On groas

Plug ChatGPT, Claude or any AI agent into groas and let it do the work of a whole marketing team: $19 a month flat with every feature included, and autopilot whenever you’re ready.

[Start For $19](https://groas.com/pricing)

[Or have groas run it for you](https://groas.com/done-for-you)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is the marketing infrastructure for your own AI: it connects your AI to all your marketing and gives it proprietary marketing data collected 24/7. Plug Claude, ChatGPT or any AI agent into groas and run all your marketing from it, for a tiny fraction of the usual cost, with the groas app there whenever you want to see more. It is the layer ChatGPT, Claude and any AI assistant run a business’s online marketing on. Through groas an assistant can do anything you could do natively in Google Ads, Bing Ads, Meta Ads and ChatGPT ads, on your website and in AI search, and, where the owner switches it on, can run connected campaigns by itself, so AI agents do the work a whole marketing team used to. groas also gives those agents large-scale proprietary marketing data it collects around the clock, on any schedule the owner chooses: every ad competitors run on Google and Meta, their landing pages and full funnels with the offers and prices in them, Reddit customer research, AI search answers from ChatGPT, Google AI Overviews, Gemini, Claude and Perplexity, and SEO data from Semrush and Ahrefs, backlinks included ($19 a month, flat, with everything included, for unlimited teammates on one shared workspace and unlimited ad accounts and websites, delivered as an MCP server). Separately, its done-for-you service has senior people and proprietary quantitative systems run paid ads, SEO and AI search for businesses, and white label for agencies, from $5,000 a month.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#webpage","url":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a","name":"675 Live-Answer Fetches of robots.txt. Still Not a Recommendation.","description":"The groas snapshot records 675 live-answer fetches for robots.txt and 387 for an article. Here’s what those counts prove—and what requires saved answers.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/bb2741c5-4a07-4dbd-b2e2-a1ebe0be6f10.png"},"breadcrumb":{"@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#breadcrumb"},"dateModified":"2026-11-17T05:14:22.750Z","mainEntity":{"@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"675 Live-Answer Fetches of robots.txt. Still Not a Recommendation.","item":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#article","url":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a","mainEntityOfPage":{"@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"675 Live-Answer Fetches of robots.txt. Still Not a Recommendation.","description":"The groas snapshot records 675 live-answer fetches for robots.txt and 387 for an article. Here’s what those counts prove—and what requires saved answers.","datePublished":"2026-11-17T05:14:22.702Z","dateModified":"2026-11-17T05:14:22.750Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/bb2741c5-4a07-4dbd-b2e2-a1ebe0be6f10.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"fetches, live-answer fetches, robots txt, live-answer, answer, answers, robots, txt","wordCount":1898,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/675-live-answer-fetches-of-robots-txt-still-don-t-prove-a#faq","mainEntity":[{"@type":"Question","name":"Does a live-answer fetch mean AI tools are recommending my business?","acceptedAnswer":{"@type":"Answer","text":"No. A live-answer fetch only proves that an AI crawler retrieved a page while an answer was being generated. It does not prove the page was cited, linked to, or recommended in the finished answer. Recommendation requires evidence from the answer itself, such as saved answer text and its displayed sources."}},{"@type":"Question","name":"What is the difference between hits, verified hits and live-answer fetches in AI crawler reports?","acceptedAnswer":{"@type":"Answer","text":"A hit is a request arriving with a claimed AI user-agent, which does not prove the bot is genuine. A verified hit passed an IP-range or reverse-DNS identity check. A live-answer fetch is a verified request associated with live-answer generation. None of the three proves that a page was cited in an answer or that the business was recommended."}},{"@type":"Question","name":"Why would robots.txt get more live-answer fetches than an actual article?","acceptedAnswer":{"@type":"Answer","text":"Because AI agents fetch the crawling rulebook as part of access checks that accompany retrieval, and different agents do different jobs. PerplexityBot follows robots.txt, for example, while Perplexity-User generally ignores it because a person requested the page. Those fetches concern access instructions, not content selection, so they say nothing about whether the business was recommended."}},{"@type":"Question","name":"Why should I verify AI bot requests before counting them?","acceptedAnswer":{"@type":"Answer","text":"Because a user-agent string is a claim, not identification. In the snapshot described, the Google Ads article kept roughly 98 percent of its claimed requests through verification, while the RSS endpoint kept under 1 percent. Unverified requests should not be counted as confirmed AI activity, so verification changes what the numbers actually mean."}},{"@type":"Question","name":"If my homepage has hundreds of AI fetches, does that mean ChatGPT recommends me?","acceptedAnswer":{"@type":"Answer","text":"No. The homepage can record the most live-answer fetches and still not establish what any answer said about the business. A mention, a citation and a recommendation are separate outcomes, and a homepage fetch proves none of them on its own. Until the saved answer is available, call the total retrieval rather than visibility won."}},{"@type":"Question","name":"A page is missing from my AI visibility report, so should I publish more content on that topic?","acceptedAnswer":{"@type":"Answer","text":"Not on that evidence alone. If a page does not appear in the report's path list, the only supported conclusion is that it is absent from the displayed rows. It might have no fetches or activity below the display limit. Request the complete export before commissioning content to fix a supposed retrieval gap."}},{"@type":"Question","name":"Can I calculate a citation rate by dividing citations by fetches?","acceptedAnswer":{"@type":"Answer","text":"No. Fetches count retrieval events, and the same answer may involve repeated requests across unknown prompts, models and days, so the units do not match. A proper citation rate is answers linking to the domain divided by answers checked, tracked per model over time with fixed prompts. Fetch totals are better used to prioritize which pages to test."}}]}]}
```
