Claude-User Fetched Our robots.txt 1,071 Times: An AI Citation Checklist That Starts With Access

Claude-User fetched our robots.txt 1,071 times during live-answer requests in 30 days. That is where I start an AI citation checklist: not with better prose, but with whether an assistant can get through the door. Our edge logs show the order worth checking: access first, extractability second, content third. If permission or delivery fails, the page you spent a week polishing is beside the point.

1. Read the access numbers before editing a page

Here is the month from groas’s own edge logs: one domain, one 30-day window, no panel and no extrapolation. The robots.txt monitoring supplies the first row; our ChatGPT fetch log supplies the second.

The 577 is fetches, not 577 distinct answers or citations. It is roughly 64% of the changelog page’s verified fetches. The page is dated, specific, and built to record what changed in Google Ads in 2026 and when. That makes it useful material for an answer, though these logs alone cannot tell us every answer in which it was cited. We did not keep a separate raw count for that page, so the table does not pretend otherwise.

The other limit matters more: raw bot hits prove little. Of 15,880 requests for robots.txt, only 3,947 verified against published IP ranges. A week-long log study across four small sites found 287 requests from one hosting network claiming eight bot identities. Count user-agent strings without verification and you may be counting costumes. Use the verified, live-answer subset to decide what to investigate next.

2. Check access before you check copy

Work these in order. A blocked fetcher cannot admire your new heading, and a sitemap full of dead ends is not doing it any favors.

• Check robots.txt for each fetcher by name. OpenAI distinguishes GPTBot, which trains models; OAI-SearchBot, which supports ChatGPT search; and ChatGPT-User, which fetches because a person asked a question (OpenAI bot docs). Do not treat a rule for one as a rule for all three. Anthropic distinguishes ClaudeBot, Claude-SearchBot, and Claude-User, and says all three honor robots.txt. If you want user-directed Claude fetches, check the rule for Claude-User. Google-Extended is an opt-out token for Gemini training, not a crawler or an AI Overviews switch.

• Check whether the WAF blocks verified live-answer fetchers. A robots.txt rule cannot rescue a request stopped at the edge. Cloudflare’s AI Crawl Control can place Claude-User in an “AI Crawler” category, so inspect category-level blocks rather than assuming they affect training crawlers alone. Perplexity says its user fetcher generally ignores robots.txt, while OpenAI says robots.txt rules may not apply to ChatGPT-User because a person requested the fetch. Audit robots.txt and WAF behavior separately, matching user agent with a published IP range before calling a request verified.

• Keep the sitemap current and limited to pages you would want quoted. Redirects, 404s, tag archives, and login walls do not become good answer pages because they appear in a sitemap. If a page has no useful answer for a buyer, leave it out.

After changing robots.txt, give the change time to take effect. OpenAI and Perplexity say updates can take up to 24 hours to propagate. Edit it today; test again tomorrow, not in ten minutes.

3. Make the answer extractable

Permission gets a fetcher to the page. It still needs an answer in the HTML it receives. This is where a page can look fine to a person and give an assistant very little to work with.

• Put the core answer in server-rendered HTML. Do not depend on a click, an interaction-loaded accordion, or a JavaScript bundle to supply the sentence you want quoted. A blocked resource or script error can leave that sentence out of the fetched response. Check page source, not only the rendered DOM in DevTools, and search for the exact wording. If it is absent from the raw HTML, a fetch of that HTML cannot lift it. This is the sort of failure GEO Toolbox identifies among pages that rank but go uncited.

• Answer the buyer’s question in the first 100 words under a clear heading. Give the passage enough context to stand alone before moving into caveats and detail. Do not make a fetcher assemble the answer from three sections. Our changelog follows a simple pattern: dated heading, change, detail. It drew 577 live-answer fetches from ChatGPT during the window, although that count does not prove the heading caused them.

• Skip llms.txt as a substitute for readable pages. A June 2026 study across 2,400 tracked domains found that publishing it moved citation rates by 0.2 percentage points, less than the untouched control group; crawler logs in the same discussion showed AI bots rarely fetching the file. Do the HTML check first. A file the assistant does not read will not repair an answer it cannot extract.

The test here is plain: can you find the complete answer in the fetched HTML? If not, fix delivery before rewriting the answer.

4. Give the assistant something worth quoting

Once access and extraction work, content finally gets its turn. The question is not whether the page sounds polished. It is whether its passage gives an assistant a specific, usable reason to cite it instead of another source.

• Show a visible last-updated date. A ConvertMate study of 80 million citations found that content updated within the last 30 days earned 3.2 times more citations than older content. That is a study finding, not a promise for your page. A CMS refresh that leaves an old date in the visible text does not help a reader judge when the information was current. Date the material they can see, and update the substance when it changes.

• Use specific numbers, named entities, and clear claims. The distinction is between commodity content that could have come from anyone and non-commodity material with something concrete to report. A discussion of Google’s guidance describes a robot-vacuum blog appearing ahead of the NYT in AI answers because its reviews contained specific numbers. Do not manufacture a number to imitate that. If you have a measured result, a named method, or a dated change, state it plainly.

• Give each page one primary buyer intent. A page trying to answer what a service costs, how it works, and who it suits can leave each answer too thin to use. Pick the question the page is for, answer it directly, and let supporting detail serve that answer. This is usually cheaper than commissioning another broad category page.

Ranking is not the same outcome as being cited. In a dataset of 645 AI-services queries across three markets, 50% of cited domains did not appear in the organic top 10 for the same query. That finding does not make rankings irrelevant. It does mean a source slot is worth checking on its own terms: can the assistant find a self-contained, dated answer it has reason to use?

5. Check whether other sources agree with you

Your page is not the only account an assistant can read. If your site describes the business one way and third-party records describe it another, tightening a paragraph on your site will not remove the contradiction.

• Check that important business claims have support outside your own site. If you call yourself the leading provider of a service but directories, reviews, and industry publications say nothing close, do not expect that self-description to carry the whole answer. Claim and complete relevant profiles and listings, including your Google Business Profile and the industry directories where your competitors appear. Make the record inspectable; do not treat this as a license to repeat an unsupported superlative everywhere.

• Keep your name, category, location, dates, and methods consistent. Xpand Digital calls this checkable consistency. If your site gives one founding year and LinkedIn gives another, a reader has a reason to hesitate. Check the details across sources, not just the copy on the page you hope will be cited.

The places to check may not be the ones your content calendar favors. In the dataset of 645 AI-services queries, the most-cited domains included YouTube, LinkedIn, Reddit, and Quora. In a Clearbrand study of 453 buyer-intent SaaS keywords, YouTube received 20.9% of AI Overview citations. Those are different samples, not a universal source ranking. They are enough reason to inspect whether a relevant video, podcast appearance, or knowledgeable forum answer belongs in your category’s record.

6. Verify the fetches and the answers

Everything above is a hypothesis until you check what happened. Keep fetch verification and answer checking separate: one tells you whether a claimed bot reached you; the other tells you whether an assistant used you in a response.

• Report verified live-answer fetches, not raw “AI bot traffic.” In our 30-day robots.txt log, 15,880 raw hits narrowed to 3,947 verified requests and 1,071 Claude-User live-answer fetches. Those are three different numbers. An academic analysis of scraper compliance describes traffic concentration within one ASN as a useful spoofing clue, but a clue is not IP verification. Match the user agent and published IP range, then separate user-initiated fetches from other crawler traffic.

• Re-test target prompts monthly and log the citations. Keep a fixed set of 10 to 20 core buyer questions. Run them in ChatGPT, Claude, Perplexity, and Google AI Mode on a schedule. Record whether you appear, who appears instead, and whether a cited source supports the claim. GEO Toolbox recommends this fixed-prompt approach. Prompt results show what those checks returned; edge logs help establish whether fetches reached your site. Neither is a substitute for the other.

If you want a shorter pre-publish pass alongside these checks, use our checklist for pages you want AI search to cite. If you would rather have someone run the technical and visibility work continuously, groas’s SEO & AI Search service covers technical fixes, new pages, and visibility work across AI search for a flat monthly fee.

The last check is the one teams skip

If I keep only one recurring item, it is this: re-test your target prompts monthly and log what you see. Our robots.txt analysis showed why raw traffic can mislead; our ChatGPT fetch log showed where live-answer requests reached specific pages. Neither replaces checking the answers themselves.

Skip that last item and a missing citation becomes an expensive guess. You rewrite the page, change the headings, and commission more copy while a WAF rule may still be stopping Claude-User at the edge. The cost is not just the rewrite. It is another month of not knowing which check failed.