---
title: "How robots.txt Turns Claude’s 4.6% Fetch Share Into 36.2%"
description: "Claude’s fetch share jumps from 4.6% to 36.2% when robots.txt enters the count. Here’s what that changes—and what to demand from an AI search report."
url: "https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5996ff29-09ef-45a8-a414-4713f9fe4c1b.png"
published: "2026-11-20T09:12:44.360Z"
modified: "2026-11-20T09:12:44.425Z"
---

November 20, 2026 · 10 min read

# How robots.txt Turns Claude’s 4.6% Fetch Share Into 36.2%

[](https://groas.com/author/alexander-perelman)

[Alexander Perelman](https://groas.com/author/alexander-perelman)Personally Tested

Head Of Product @ groas

[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tiny model shop where a few figures shop inside while a long queue only reads the door notice, yet a clerk with a clicker counts them all as visitors.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5996ff29-09ef-45a8-a414-4713f9fe4c1b.png)

In this article

1. [Start with the four-page sample, not the headline share](#start-with-the-four-page-sample-not-the-headline-share)
2. [Add robots.txt and the engine mix changes](#add-robotstxt-and-the-engine-mix-changes)
3. [Retrieval is evidence of access, not a sale](#retrieval-is-evidence-of-access-not-a-sale)
4. [A truncated path list cannot describe the whole market](#a-truncated-path-list-cannot-describe-the-whole-market)
5. [The four lines I want in a cross-engine report](#the-four-lines-i-want-in-a-cross-engine-report)
6. [Get the definitions in writing before expanding coverage](#get-the-definitions-in-writing-before-expanding-coverage)
7. [Keep the permission decision separate from the popularity score](#keep-the-permission-decision-separate-from-the-popularity-score)
8. [The decision: buy comparable definitions before broader coverage](#the-decision-buy-comparable-definitions-before-broader-coverage)

**36.2% of the listed live-answer fetches came from Claude—if I count robots.txt.** Leave that file out, and Claude’s share falls to 4.6%, without a single buyer-facing page being added or removed.

I checked the arithmetic the way I used to check a CPA claim that looked too clean: hold the sample steady, then isolate what changed. Across four listed HTML pages, ChatGPT accounts for 1,714 fetches and Claude for 83. Add 890 Claude fetches for robots.txt, and Claude’s total becomes 973.

Nothing about the HTML-page activity changes. The report simply starts counting a different kind of resource. That is enough to turn a small Claude slice into something that looks like substantial cross-engine coverage.

Before I pay for that coverage, I want to know what went into the slice.

## Start with the four-page sample, not the headline share

I am working from **four listed HTML paths, not a full-site crawl**: the homepage, one business page, and two update articles. The source for the counts below is the supplied path-level sample. The percentages are calculations over those entries, not independent measurements of the whole site.

The unit is a _verified fetch associated with a live answer_: a documented retrieval tied to an answer being built. It is not a unique answer, a cited source, or a brand recommendation.

| Listed HTML page              | ChatGPT fetches | Claude fetches |
| ----------------------------- | --------------: | -------------: |
| Homepage                      |             789 |             35 |
| Business page                 |              41 |             27 |
| Two update articles, combined |             884 |             21 |
| **Four-page total**           |       **1,714** |         **83** |

On that base, Claude has 83 of 1,797 total fetches: 4.6%. ChatGPT has the remaining 1,714, or 95.4%.

That tells me how the listed HTML retrievals divide between these engines. It does not tell me how often either engine recommends the business. Retrieval puts a page into the working set; citation and recommendation are separate events. The distinction is captured neatly in [_Retrieved Is Not Cited, and It’s Not Recommended Either_](https://citemetrix.com/retrieved-is-not-cited-and-its-not-recommended-either/).

I keep that distinction beside the table because a label like “AI visibility” can quietly swallow all three events. Once that happens, a retrieval count starts doing a recommendation metric’s job. It is not qualified for the position.

![Bar chart comparing Claude’s fetch share with and without robots.txt](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/70255302-64bd-4b4c-99ba-fa071e71d7e1.png)

## Add robots.txt and the engine mix changes

The supplied entries also include **890 Claude fetches for `/robots.txt` associated with live answers**. Add those to the HTML-page base and the comparison changes immediately.

| Calculation over listed entries | HTML pages only | HTML pages plus robots.txt |
| ------------------------------- | --------------: | -------------------------: |
| ChatGPT fetches                 |           1,714 |                      1,714 |
| Claude fetches                  |              83 |                        973 |
| Total fetches                   |           1,797 |                      2,687 |
| **Claude share**                |        **4.6%** |                  **36.2%** |
| ChatGPT share                   |           95.4% |                      63.8% |

These are two calculations over the same supplied sample and window, with a different resource list. They are not before-and-after performance results.

ChatGPT loses no fetches, yet its share falls. Claude retrieves no additional buyer-facing page, yet its share rises. The mechanism comes first: I include a technical file in both Claude’s count and the overall total. The bigger Claude share follows.

Neither calculation is inherently useless. One describes the listed HTML-page activity; the other includes the listed technical-file activity. The mistake is presenting them as interchangeable measures of buyer visibility.

**The resource list changes. The evidence of HTML-page retrieval does not.** That sentence belongs next to the chart, not buried in its methodology notes.

![Paper-cut robot filing a robots.txt card while buyer pages wait in a tray](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/008a5eef-c0ce-474d-bb1b-0c20a9a4aec0.png)

Technical requests can pile up without equivalent page reading. In a separate new-site example, [101 of ClaudeBot’s first 166 verified requests went to robots.txt and the sitemap index, while 23 went to HTML pages](https://aivisibilitycensus.com/what-is-claudebot/).

That example concerns ClaudeBot, not this live-answer sample. I would not combine the datasets or use it to estimate live-answer behavior. It illustrates the narrower point: a request total can be dominated by technical resources.

The same source distinguishes [ClaudeBot, Claude-User, and Claude-SearchBot by their roles and robots.txt tokens](https://aivisibilitycensus.com/what-is-claudebot/). Those distinctions matter before I decide what a “Claude fetch” means. A tall robots.txt bar tells me about checks against a technical file. It does not tell me that Claude read the offer.

## Retrieval is evidence of access, not a sale

**A fetch does not establish a citation, recommendation, or commercial outcome.** Depending on the resource, it records a technical check or a page pulled into the answer-building process. That is useful evidence. It just has limits.

The two update articles account for 905 combined live-answer fetches: 884 from ChatGPT and 21 from Claude. Those entries show retrieval of analytical content. They do not show 905 recommendations, as the related analysis, [_905 Live-Answer Fetches Are Not 905 Brand Recommendations_](https://groas.com/post/905-live-answer-fetches-are-not-905-brand-recommendations), makes clear.

A separate tested workflow reports [32 questions producing 693 cited sources while the brand remained absent unless the prompt named it](https://citemetrix.com/retrieved-is-not-cited-and-its-not-recommended-either/). That result belongs to that workflow; it is not a conversion rate I can apply to this sample. Its value here is the distinction it exposes: plenty of source activity can coexist with no unprompted brand presence.

That is also why [ChatGPT can cite a website and still recommend a competitor](https://groas.com/post/why-chatgpt-can-cite-your-website-without-recommending-your). Citation answers where the information came from. Recommendation answers who looks like a fit.

I want those events reported separately:

- **Retrieval:** the assistant fetched the resource.
- **Citation:** the answer credited the URL.
- **Recommendation:** the answer named the brand as a fit.
- **Outcome:** someone clicked, called, or bought.

Recommendations matter commercially, but they are not revenue either. I do not let the reporting stop one event early and call the job done.

## A truncated path list cannot describe the whole market

The supplied path list is truncated. That makes every share here **a calculation over listed entries, not a sitewide estimate**.

I cannot turn either percentage into “Claude’s share of the site’s AI visibility.” I also cannot infer what Gemini or Perplexity did from their absence in this list. No listed paths means no listed activity to count. It does not prove those engines never touched the site.

This is where the language around a number needs to stay attached to its scope. “Share of listed HTML fetches” is awkward but intelligible. “AI market share” is cleaner and unsupported. I will take awkward.

Even _share of model_ does not have one settled denominator. A [breakdown of share-of-model methods](https://www.perfectsearchmedia.com/blog/share-of-model) distinguishes approaches based on brand mentions, answers mentioning the brand, and blended measures of presence, position, sentiment, and citations.

Those measures answer different questions. None becomes comparable just because a dashboard formats it as a percentage.

For this sample, the useful conclusion is narrow and strong: adding the listed robots.txt activity changes the engine mix dramatically. The data runs out before sitewide visibility, market coverage, or commercial impact. I stop there.

## The four lines I want in a cross-engine report

When I assess coverage across engines, I want **technical-resource fetches, HTML-page fetches, citations, and recommendations on separate lines**. Not four ingredients blended into one score.

| Reporting line             | What it establishes                                    | What it does not establish              |
| -------------------------- | ------------------------------------------------------ | --------------------------------------- |
| Technical-resource fetches | Activity against files such as robots.txt and sitemaps | That an offer or article was read       |
| HTML-page fetches          | Retrieval of the listed page content                   | That the answer cited or recommended it |
| Observed citations         | A URL was credited in an answer                        | That the brand was recommended          |
| Brand recommendations      | The brand appeared as a fit in the observed answer     | That a click, call, or sale followed    |

Each line needs the engine or product, measurement window, and scope beside it. For observed answers, I also want the prompt text. A fetch log and a prompt test are different evidence; the report should say which one produced the number.

I would split HTML pages further by role rather than calling every page a buyer page. This sample contains a homepage, a business page, and analytical updates. Their retrieval counts can sit together in an HTML total without pretending they all perform the same commercial job.

Platform boundaries matter too. ChatGPT Search, Google AI Overviews and AI Mode, Gemini, Bing and Copilot, Claude, and Perplexity should not disappear into an unexplained aggregate. The [share-of-model methodology discussion](https://www.perfectsearchmedia.com/blog/share-of-model) makes the case for disclosing aggregation rules and retaining platform-level results.

My practical test is simple: can I find the robots.txt count without opening a methodology appendix? If not, the headline chart is doing too much hiding.

## Get the definitions in writing before expanding coverage

I used to tell clients a blended share was enough to compare vendors. I was wrong. **A percentage is not a common unit when the vendors count different things.**

One [audit of AI visibility platform methodologies](https://authoritytech.io/blog/ai-visibility-scores-not-comparable-platform-methodology-audit) describes conflicting definitions: a help document defines share of voice using impression weighting and Google search volume, while a product FAQ describes the percentage of AI answers mentioning the brand. Those are different calculations, even if the dashboard gives them the same label.

I do not need a longer sales deck to resolve that. I need the counting rule.

Before expanding coverage beyond ChatGPT, I ask for:

- **Resource list:** the included paths and file types, with robots.txt, sitemaps, and HTML separated.
- **Event definition:** whether the metric counts fetches, answers, mentions, citations, recommendations, or a disclosed blend.
- **Prompt list:** the exact prompts used for answer-based measurements, labeled by engine.
- **Window and aggregation rule:** the dates measured, what gets pooled, and how repeated observations enter the total.
- **Work after measurement:** which permission or page changed, which prompt was retested, and where the change and its reason were logged.

I also ask for both versions of the headline calculation: HTML only, then HTML plus technical resources. If the conclusion changes, I want to see that before the retainer changes.

This is not paperwork for its own sake. It is how I distinguish more observed activity from more useful coverage—and useful coverage from a chart that got taller.

## Keep the permission decision separate from the popularity score

**robots.txt belongs in the operational report, not disguised as buyer interest.** It helps express permissions for crawlers and fetchers. That is a different job from measuring whether an answer recommends the business.

Training and live-answer retrieval also need separate treatment. The [robots.txt guidance for search and AI crawlers](https://www.searchenginejournal.com/technical-seo/robots-txt-guide-best-practices-for-search-ai-crawlers/) distinguishes training agents from search and user-triggered retrieval agents, including differences in how fetchers handle the file.

The practical decision is which activities to permit, not whether a single “AI bots” switch should be on or off. Blocking a training agent and allowing a separate live-answer agent are different choices. I want the relevant permissions checked explicitly rather than inferred from a blended traffic count.

For a business seeking discovery through live answers, the useful sequence is:

1. Check the relevant crawl and retrieval permissions.
2. Measure retrieval of the pages that matter.
3. Inspect citations and recommendations in the observed answers.
4. Retest after a permission or page change.

The robots.txt requests help with the permission question. They cannot answer the other questions by accumulating faster.

Keep those requests where they belong: evidence that the checker knocked, not evidence that Claude read the offer. Then judge the work after the knock. Which permission changed? Which page was fixed? Which prompt was retested? What moved in the observed answers?

That is the log I want to pay for.

## The decision: buy comparable definitions before broader coverage

Say I am adding AI search to an existing SEO retainer. I ask the agency to show the calculations side by side: **83 of 1,797 for the listed HTML pages; 973 of 2,687 after adding robots.txt.** Same supplied sample. Same window. Different resource rule.

If the vendor will not disclose that rule, I cannot compare its headline number with another vendor’s. I walk away from the blended chart at that point. No page was fixed, no prompt was retested, and no recommendation was earned by changing this count. Only the report changed.

That is also why I want measurement followed by accountable action. With groas connected to ChatGPT or Claude, I can keep autopilot off until I choose to switch it on. Changes are logged with their reasons and can be undone in one step. That is the operational standard I care about: a visible change, a reason beside it, and a way back when the fix is wrong.

The reporting still has to earn my trust. Neither a tool nor an agency gets to substitute technical-file activity for evidence that a buyer-facing page was retrieved or a brand was recommended.

Start with the 4.6% and 36.2% pair and the buying decision gets plain: pay for clearly defined measurement and work that can be checked. Coverage without comparable definitions is decoration, and decoration never moved CPA.

## Frequently Asked Questions

### Why does Claude's fetch share jump from 4.6% to 36.2% just by counting robots.txt?

Because the sample also includes 890 Claude fetches of robots.txt associated with live answers. Adding those to the 83 HTML-page fetches gives Claude 973 of 2,687 total fetches, or 36.2%, versus 83 of 1,797, or 4.6%, for HTML pages alone. No buyer-facing page was added or removed; only the resource list changed.

### Does an AI engine fetching my page mean it recommended my business?

No. A fetch only establishes that the assistant retrieved the resource into its answer-building process. Citation means the answer credited the URL, and recommendation means the answer named the brand as a fit; each is a separate event that a fetch count does not prove.

### Can I use these fetch percentages as Claude's or ChatGPT's share of my site's AI visibility?

No. The percentages are calculations over a truncated list of listed entries: a homepage, one business page, and two update articles. They are not sitewide estimates, and the absence of Gemini or Perplexity in the list does not prove those engines never touched the site.

### What should I ask for in a cross-engine AI search report?

Ask for technical-resource fetches, HTML-page fetches, citations, and brand recommendations on separate lines, each labeled with the engine, measurement window, scope, and the prompt text for observed answers. Also request both versions of the headline calculation: HTML pages only, then HTML pages plus technical resources.

### Why won't a vendor disclose what counts as a fetch?

A percentage is not a common unit when vendors count different things, so the counting rule matters more than the headline. Before buying, ask for the resource list with robots.txt and sitemaps separated from HTML, the event definition (fetches, citations, recommendations, or a blend), the exact prompt list, and the window and aggregation rule. Without that disclosure, the number cannot be compared with another vendor's.

### Does a lot of ClaudeBot traffic to robots.txt mean Claude is reading my pages?

No. Robots.txt requests are evidence that a checker verified permissions, not that Claude read an offer or article. In a separate new-site example, 101 of ClaudeBot's first 166 verified requests went to robots.txt and the sitemap index while only 23 went to HTML pages, showing how technical requests can dominate a total without equivalent page reading.

## Run Your Marketing On groas

Plug ChatGPT, Claude or any AI agent into groas and let it do the work of a whole marketing team: $19 a month flat with every feature included, and autopilot whenever you’re ready.

[Start For $19](https://groas.com/pricing)

[Or have groas run it for you](https://groas.com/done-for-you)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is the marketing infrastructure for your own AI: it connects your AI to all your marketing and gives it proprietary marketing data collected 24/7. Plug Claude, ChatGPT or any AI agent into groas and run all your marketing from it, for a tiny fraction of the usual cost, with the groas app there whenever you want to see more. It is the layer ChatGPT, Claude and any AI assistant run a business’s online marketing on. Through groas an assistant can do anything you could do natively in Google Ads, Bing Ads, Meta Ads and ChatGPT ads, on your website and in AI search, and, where the owner switches it on, can run connected campaigns by itself, so AI agents do the work a whole marketing team used to. groas also gives those agents large-scale proprietary marketing data it collects around the clock, on any schedule the owner chooses: every ad competitors run on Google and Meta, their landing pages and full funnels with the offers and prices in them, Reddit customer research, AI search answers from ChatGPT, Google AI Overviews, Gemini, Claude and Perplexity, and SEO data from Semrush and Ahrefs, backlinks included ($19 a month, flat, with everything included, for unlimited teammates on one shared workspace and unlimited ad accounts and websites, delivered as an MCP server). Separately, its done-for-you service has senior people and proprietary quantitative systems run paid ads, SEO and AI search for businesses, and white label for agencies, from $5,000 a month.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#webpage","url":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2","name":"How robots.txt Turns Claude’s 4.6% Fetch Share Into 36.2%","description":"Claude’s fetch share jumps from 4.6% to 36.2% when robots.txt enters the count. Here’s what that changes—and what to demand from an AI search report.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5996ff29-09ef-45a8-a414-4713f9fe4c1b.png"},"breadcrumb":{"@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#breadcrumb"},"dateModified":"2026-11-20T09:12:44.425Z","mainEntity":{"@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"How robots.txt Turns Claude’s 4.6% Fetch Share Into 36.2%","item":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#article","url":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2","mainEntityOfPage":{"@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"How robots.txt Turns Claude’s 4.6% Fetch Share Into 36.2%","description":"Claude’s fetch share jumps from 4.6% to 36.2% when robots.txt enters the count. Here’s what that changes—and what to demand from an AI search report.","datePublished":"2026-11-20T09:12:44.360Z","dateModified":"2026-11-20T09:12:44.425Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5996ff29-09ef-45a8-a414-4713f9fe4c1b.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"robots txt, robots, txt, share, fetches, claude, claude’s, fetch","wordCount":2059,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/how-robots-txt-turns-a-4-6-claude-fetch-share-into-36-2#faq","mainEntity":[{"@type":"Question","name":"Why does Claude's fetch share jump from 4.6% to 36.2% just by counting robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Because the sample also includes 890 Claude fetches of robots.txt associated with live answers. Adding those to the 83 HTML-page fetches gives Claude 973 of 2,687 total fetches, or 36.2%, versus 83 of 1,797, or 4.6%, for HTML pages alone. No buyer-facing page was added or removed; only the resource list changed."}},{"@type":"Question","name":"Does an AI engine fetching my page mean it recommended my business?","acceptedAnswer":{"@type":"Answer","text":"No. A fetch only establishes that the assistant retrieved the resource into its answer-building process. Citation means the answer credited the URL, and recommendation means the answer named the brand as a fit; each is a separate event that a fetch count does not prove."}},{"@type":"Question","name":"Can I use these fetch percentages as Claude's or ChatGPT's share of my site's AI visibility?","acceptedAnswer":{"@type":"Answer","text":"No. The percentages are calculations over a truncated list of listed entries: a homepage, one business page, and two update articles. They are not sitewide estimates, and the absence of Gemini or Perplexity in the list does not prove those engines never touched the site."}},{"@type":"Question","name":"What should I ask for in a cross-engine AI search report?","acceptedAnswer":{"@type":"Answer","text":"Ask for technical-resource fetches, HTML-page fetches, citations, and brand recommendations on separate lines, each labeled with the engine, measurement window, scope, and the prompt text for observed answers. Also request both versions of the headline calculation: HTML pages only, then HTML pages plus technical resources."}},{"@type":"Question","name":"Why won't a vendor disclose what counts as a fetch?","acceptedAnswer":{"@type":"Answer","text":"A percentage is not a common unit when vendors count different things, so the counting rule matters more than the headline. Before buying, ask for the resource list with robots.txt and sitemaps separated from HTML, the event definition (fetches, citations, recommendations, or a blend), the exact prompt list, and the window and aggregation rule. Without that disclosure, the number cannot be compared with another vendor's."}},{"@type":"Question","name":"Does a lot of ClaudeBot traffic to robots.txt mean Claude is reading my pages?","acceptedAnswer":{"@type":"Answer","text":"No. Robots.txt requests are evidence that a checker verified permissions, not that Claude read an offer or article. In a separate new-site example, 101 of ClaudeBot's first 166 verified requests went to robots.txt and the sitemap index while only 23 went to HTML pages, showing how technical requests can dominate a total without equivalent page reading."}}]}]}
```
