A money page with no AI crawler visits deserves an access check before another rewrite. Check the fetch first; work on earning the citation second.

I spent years filtering bots out of logs. Humans were signal; bots were noise. AI search makes that distinction less useful. Requests from OAI-SearchBot, ClaudeBot or PerplexityBot can help show whether crawlers reach the pages you want answers to cite. But a crawl proves only that a bot requested a URL, not that an answer cited it.

By the end, you will have a 30-day record of crawler visits, a list of important pages missing from that record, and one access problem fixed or ruled out.

Before starting, get:

  • Server access logs or CDN logs covering the last 30 days.
  • Permission to inspect robots.txt and firewall rules.
  • A list of your top 20 money pages.
  • Optionally, groas connected to your assistant, with the relevant log export available for analysis.

I used to tell clients to rewrite pages first. I was wrong. A better paragraph does not fix a 403.

Step 0: Separate search crawlers from training crawlers

Action: Build your filter around these names:

NameWhat to distinguish
GPTBotOpenAI crawling for potential model training
OAI-SearchBotOpenAI crawling for ChatGPT search
ChatGPT-UserFetches triggered by a user’s request
ClaudeBot, anthropic-aiAnthropic-related names to inspect
PerplexityBotPerplexity crawler requests
Bytespider, CCBotAdditional names to include for completeness

OpenAI separates training, search and user-triggered fetches. Training and search have independent robots.txt groups, so blocking GPTBot does not require blocking OAI-SearchBot.

Keep training access separate from search access. If your goal is visibility in live answers, a training-crawler count is not a substitute for inspecting search and on-demand requests.

Treat Google-Extended separately. It is a robots.txt control, not a crawler user-agent, so searching access logs for it will not produce a useful visit count. Crawling happens under normal Google user-agents. The control concerns use for Gemini training and grounding, not inclusion in AI Overviews.

Expected result: A list of actual user-agent strings to search, plus a separate Google-Extended policy check.

Mistake to avoid: Calling everything AI traffic. A missing training crawl and a missing search fetch are not the same finding. For the mechanics, see what actually hits your server when ChatGPT looks for an answer.

Step 1: Pull 30 days of requests, not an analytics report

Action: Export the last 30 days of server or CDN access logs. Include the timestamp, user-agent, source IP, requested path and response status.

Skip GA4 for this check. Crawlers can fetch HTML without running the analytics script, and Google excludes known bots automatically. Use request logs as the record of access.

On Cloudflare, open AI Crawl Control and inspect the Crawlers and Metrics views. Look for requests by crawler, status codes and requested paths.

For a plain Nginx or Apache log, filter the exported records. Replace the path below with your uncompressed 30-day export:

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|anthropic-ai|PerplexityBot|Bytespider|CCBot' /path/to/30-day-access.log

Keep the complete matching lines. You need the user-agent and IP later, not just a count of paths.

Group the results by crawler and page, then record:

  • Total requests by crawler.
  • The top 20 requested paths.
  • Status codes for each important page.
  • Important pages with no matching requests.

Read the codes before celebrating the volume. 2xx means the request was served; 403 means forbidden; 404 means missing. A crawler retrying a broken URL is not a content strategy.

Expected result: A table showing which claimed crawlers requested which pages and what response each received.

Mistake to avoid: Trusting the user-agent name. It is a claim, not identification. For GPTBot and OAI-SearchBot, compare the source IP with the bot’s published IP ranges. Treat unverified requests separately rather than letting them inflate your totals. High volumes of missing paths deserve inspection, not applause.

Behind a CDN, inspect which IP your origin log records. It may be the proxy rather than the visitor. Log CF-Connecting-IP or the equivalent visitor-IP field alongside the forwarded chain when needed. A hashed address cannot be matched against a published IP range afterward.

If the existing logs lack the necessary fields, note that limitation and fix collection for the next window. Do not turn incomplete history into a confident zero. For more detail, use this server-log walkthrough for ChatGPT, Claude and Perplexity requests.

Step 2: Check robots.txt and the firewall before blaming content

Action: Open your site’s /robots.txt. Inspect the groups for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, along with rules covering your important paths.

Look for Disallow: / and narrower path restrictions. Read the rules as written, not as someone remembers configuring them.

If your intended policy is to block OpenAI training while allowing its search crawler, the relevant groups look like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Expected result: Your declared crawler policy matches your actual intention. Training blocked and search allowed can be a deliberate choice. It is not automatically a configuration mistake.

Next, inspect the firewall. In Cloudflare’s AI Crawl Control, list the crawlers set to Block and compare them with your intended policy. Cloudflare provides crawler controls and request reporting, but its dashboard only sees traffic passing through Cloudflare. Detection based on user-agent strings also needs the identity checks from Step 1.

For a basic response check, replace the example URL with the affected page:

curl -I -A "OAI-SearchBot" "https://yourdomain.com/important-page"

This checks the response to your request using that user-agent. It does not authenticate you as the crawler or prove that a genuine crawler will receive the same treatment. Compare it with actual logged requests.

Mistake to avoid: Assuming an Allow in robots.txt overrides a firewall block. Check both layers. A forgotten rule can explain the missing fetch without requiring a single sentence of new copy.

Before commissioning a rewrite, run this crawler-access check.

Step 3: Turn the log export into a page-level gap list

Action: Give your assistant the filtered log export and your top 20 money-page URLs. With groas connected over MCP, use this prompt:

Using the attached 30-day crawler-log export and our list of important URLs, build a table of crawler names, request counts, requested pages and status codes.

Separate verified crawler identities from unverified user-agent claims. List important pages with no recorded requests in this export. Flag missing fields or incomplete date coverage instead of treating them as zero visits.

Through groas, help prioritize which page to inspect first. Do not change the site yet.

Expected result: A readable request table and a gap list tied to pages that matter, not just the busiest URLs.

Connecting a website is not a substitute for supplying its server or CDN logs. Make sure the assistant has the records before asking it to explain 30 days of crawler behavior. Otherwise, the polished table is doing more work than the evidence.

The reason I use an assistant here is practical: sorting requests is mechanical; deciding which missing page matters is the useful part. groas Unlimited costs $19 a month flat for unlimited teammates, ad accounts and websites. It gives your AI the infrastructure to work on the site, rather than stopping at a report.

Mistake to avoid: Exporting a CSV, admiring it and doing nothing. A crawler report that never leads to an access check or a page decision is a souvenir.

Finish with three working lists:

  • Crawled and served.
  • Requested but blocked or missing.
  • Important, with no recorded crawl.

Keep “no recorded crawl” literal. It describes the available logs, not everything an AI system might know about the page.

Step 4: Compare crawl access with actual citations

Action: Put your top 20 money pages beside the request table. Then paste five buyer questions into ChatGPT, Claude and Perplexity. Record which answers cite your pages and which name competitors instead.

Keep the questions consistent when you repeat the check. For each page, distinguish:

  • Crawled and cited: Both a recorded fetch and an observed citation.
  • Crawled, not cited in the checked answers: Access exists; inclusion remains unproven.
  • Cited, with little recorded crawling: A citation exists, but your logs do not explain its full path.
  • No recorded crawl or citation: Inspect access before investing in a rewrite.

Requests, citations and clickthrough visits are different events. Do not use one as a stand-in for another.

Expected result: One priority page with a specific next action. A 403 on a money page gives you something concrete to fix. A served page absent from five answers gives you a different question to investigate.

Mistake to avoid: Diagnosing content quality from a crawl alone. A fetch without a citation does not prove the page lost on clarity or evidence. A citation with few recent fetches does not prove it came from old training data.

Fix demonstrated access failures first. Then investigate important pages with no recorded visits. Ignore the blog archive’s impressive bot count if it tells you nothing about buyer intent.

Server log table grouping AI crawler requests into frequently visited, blocked and unvisited pages

Step 5: Fix one access problem at a time

Action: With the relevant logs and page list available, ask your assistant through groas:

Check robots.txt and our top 20 pages for anything blocking the AI search crawlers we intend to allow. Compare the findings with the attached request logs.

Propose fixes for one priority page. Show the proposed changes and explain what each would address. Keep our training-crawler policy unchanged. Do not apply changes until I approve them.

Expected result: A short proposed change, not a site-wide permission reset. Depending on the evidence, the fix may be:

  • Removing a stale search-crawler restriction in robots.txt.
  • Changing an unintended Cloudflare crawler rule from Block to Allow.
  • Repairing a 404 or 403 on the intended page URL.

Apply firewall changes in the relevant dashboard. Approve site edits individually. Changes made through groas are logged with their reason and can be undone in one step.

Record the affected URL, the change and the date. You will need all three for the recheck. If no access problem appears, stop changing access rules. “No visits” is a finding to investigate, not permission to dismantle the firewall.

Mistake to avoid: Unblocking every crawler because a checklist says so. I used to do that. Keep training blocked if that is your policy; fix unintended search-access restrictions first.

Ink cartoon of a blocked robot at a gate with a money page waiting behind it

Step 6: Verify the fetch, then check the answer separately

Action: After 7 to 14 days, rerun the Step 1 filter for the page you changed. Compare requests before and after the recorded fix date. Ask the same five buyer questions again.

Check the results in order:

  1. Access: Does the affected URL now return the intended response?
  2. Crawler activity: Do the logs show fresh, verified requests from the relevant crawler?
  3. Citation: Does any checked answer cite the page or use its facts?

Expected result: Evidence that the access problem is resolved. Fresh 200 responses from the relevant search crawler are useful confirmation. A citation is a separate outcome, not something the access fix promises within 14 days.

If there is no new request, the wait alone does not prove the fix failed. Recheck the response and your log coverage. If requests arrive but citations do not, stop treating access as the only problem.

Mistake to avoid: Calling the whole job done after one training-crawler hit. That proves neither search-crawler access nor citation visibility. Verify the part of the chain you actually changed.

Keep the monthly check focused on the next useful fix

Some live-answer activity never appears under a named bot, so even clean logs are incomplete. Treat logs as evidence of recorded requests, not a full count of AI attention. Keep the distinction intact: a crawl is a request; a citation appears in the answer a person sees.

When repeated checks show served fetches but no citations, move beyond access. Inspect the page’s clarity, structure and evidence before assuming another crawler setting will help.

Once the access fix works, leave that page’s access rules alone. The first thing to change is your target: move the same check-and-fix loop to the next important page with a demonstrated problem. I keep this monthly because robots.txt edits, plugin updates and Cloudflare toggles drift. Same three lists. One useful fix at a time.