--- title: "The AI Crawlability Pre-Flight: 24 Checks Before You Rewrite a Page" description: "Run these 24 checks for crawler access, usable HTML, answer structure and monitoring before you blame your content for missing AI citations." url: "https://groas.com/post/the-ai-crawlability-pre-flight-24-checks" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/024727cf-c2f6-4eec-bf96-55f9a446de26.png" published: "2026-10-20T05:38:35.152Z" modified: "2026-10-20T05:38:35.216Z" --- October 20, 2026 · 10 min read # The AI Crawlability Pre-Flight: 24 Checks Before You Rewrite a Page [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![An engineer finds a padlock chained to a jet's wheel while the pilot polishes the windscreen, as polishing copy is useless if crawlers are locked out.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/024727cf-c2f6-4eec-bf96-55f9a446de26.png) In this article 1. [Access: make sure the right crawlers are allowed](#access-make-sure-the-right-crawlers-are-allowed) 2. [Retrieval: make the fetched page worth fetching](#retrieval-make-the-fetched-page-worth-fetching) 3. [Structure: give extraction a clean answer](#structure-give-extraction-a-clean-answer) 4. [Entity and trust: make the business recognizable](#entity-and-trust-make-the-business-recognizable) 5. [Monitoring: catch the next break before you rewrite](#monitoring-catch-the-next-break-before-you-rewrite) 6. [Final gate: check the edge before anything else](#final-gate-check-the-edge-before-anything-else) I once spent two weeks rewriting ad copy while the landing page threw a soft 404 for half its traffic. Rewriting a page for ChatGPT when an AI crawler cannot load it is the same mistake with a newer acronym. Most businesses invisible to AI assistants have a plumbing problem to rule out first: a robots.txt line from 2023, a CDN bot rule, or a page whose copy appears only after JavaScript runs. **Test access before you edit content.** Run the final edge gate first, then work through the checks below in failure order. Stop at the first fail, fix it, and retest. I’ve left that gate as the last item because it is the one teams most often forget, and the one that can make every other pass irrelevant. ![Wrench tightening a pipe labeled AI crawler while web pages wait downstream](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a1de3295-d232-43c6-b5d9-5564a0b5673c.png) ## Access: make sure the right crawlers are allowed 1. **Allow OAI-SearchBot on answer pages.** Check `yourdomain.com/robots.txt` for a `Disallow` that covers the URLs you want cited. [Training bots and search bots are separate agents](https://securityboulevard.com/2026/07/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking/): blocking `GPTBot` does not, by itself, block ChatGPT search when `OAI-SearchBot` remains allowed. Read the rule against an actual money-page URL, not just the homepage. A clean-looking robots file is no comfort if an old path rule still catches the pages you care about. 2. **Allow Claude-SearchBot and PerplexityBot where you want answers.** Check both tokens against the same answer-page URLs. Do not treat a training opt-out as a blanket search decision: `ClaudeBot` and `CCBot` are distinct from the search bots that may fetch content for answers. Write down which agents you intend to admit and which you intend to block. Otherwise, a broad rule inherited from an earlier AI policy can quietly answer a question nobody on the team meant to ask. 3. **Allow Googlebot on pages you want in AI Overviews.** Test the exact service, product, and location URLs, not merely the domain root. [Blocking Googlebot removes eligibility for AI Overviews and AI Mode](https://securityboulevard.com/2026/07/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking/). I check this before debating headings or schema because a stray `Disallow` makes those edits beside the point. If the intended page is blocked, fix the rule and retest that URL before moving to extraction. 4. **Make Google-Extended an explicit decision.** [Google-Extended is a control token for Gemini training and grounding, not a fetcher](https://securityboulevard.com/2026/07/every-ai-crawler-in-2026-the-reference-table-and-which-ones-your-waf-is-silently-blocking/). Decide whether to allow it for grounding or disallow it as a training opt-out; do not mistake either choice for a live Googlebot fetch test. The pass condition is that someone who owns the policy can explain the setting and its purpose. A leftover line copied from an old robots file is not a strategy. 5. **Remove login, cookie, and geo walls from public answer pages.** Fetch each page logged out and from another region. Pass only when the full answer copy is available without a click, an account, or a location-specific detour. A person may know to dismiss a banner or change regions; a fetch may never get that far. If the page must stay gated, do not count its hidden copy as material an assistant can quote from the public URL. 6. **Handle user-triggered fetches at the edge.** Know which requests your robots policy does not govern. [Perplexity-User generally ignores robots.txt by design, and ChatGPT-User may also ignore it for user-triggered fetches](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). If you intend to stop those requests, use WAF or edge enforcement rather than adding another `Disallow`. If you intend to admit them, verify what page they receive. Keep that choice separate from your policy for search and training bots. ## Retrieval: make the fetched page worth fetching 7. **Put the core copy, price, and H1 in raw HTML.** View Source should show complete, readable sentences rather than an empty `
` waiting for a browser to build the page. In [a 41-day test, crawlers fetched JavaScript files but did not execute them, and discovered no pages behind JS-only links](https://speedvitals.com/blog/csr-vs-ssr-ai-seo/). Ship answer pages with SSR or SSG so the fetch contains the answer. A page that looks complete in your browser can still arrive as a shell. 8. **Check the final URL for a real 200 and a real page.** Request the live URL after every template push. Pass when it returns the intended copy with a 200 status, not an error template that also reports 200. That soft 404 is easy to miss if the test stops at the status code. I learned the expensive version in paid search: I blamed ad copy while a swapped landing-page template failed visitors. Inspect what the response says, not just what its header claims. 9. **Cut redirect chains to one 301 hop.** Follow an old URL all the way to its intended replacement and count the hops. [Single-hop 301s resubmitted with the sitemap recover fastest; chains and orphaned URLs can stall AI re-crawl for weeks](http://replatformradar.com/blog/how-ai-answer-engines-rediscover-a-site-after-a-replatform). A redirect that eventually reaches the right page still makes discovery harder than it needs to be. Fix the route before rewriting the destination page, then test both the old and final URLs. 10. **Point the canonical at the page you want cited.** On the intended answer page, check for a self-referencing canonical; on variants, check that they point to that page. Assistants may consolidate around the canonical, so a template that points every service page at a generic parent can undermine otherwise usable copy. Read the rendered tag on a money page rather than trusting a CMS setting. The pass is the correct URL in the page response, not the URL someone meant to configure. 11. **Keep the document fast and light.** Aim for a response in under about two seconds and HTML under a few hundred kilobytes. Heavy pages can time out at the edge before extraction starts. Test the document request itself rather than the time it takes your own browser to finish displaying everything. This is a rough pre-flight threshold, not a reason to strip useful copy: the question is whether the fetch reliably receives the answer before the request gives up. ## Structure: give extraction a clean answer ![Cutaway of a web page contrasting an empty JavaScript shell with full HTML and readable answer text](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/7391a542-0968-4cfa-af69-7bb8cab296dc.png) 12. **Use one H1 that names the page topic.** View Source should show one `

` identifying the service and market the page actually covers. Multiple H1s can split the extraction; a vague one makes the rest of the page work harder. Check the HTML output, since templates can add a second heading outside the editor’s view. This is not an invitation to repeat keywords. It is a check that the page announces its subject before asking an assistant to infer it. 13. **Write an H2 for each buyer question.** Use headings a reader could ask directly: cost, timeline, coverage area, or how the service works. I map prompts to sections before I write copy because a heading should tell both the person and the extractor what answer follows. Do not turn every heading into a slogan and bury the actual question beneath it. Pass when someone scanning only the H2s can find the section that addresses their question. 14. **Answer in the first two sentences under each H2.** Put the number or definition up front, then add conditions and detail. Assistants can lift the beginning of a block; they cannot quote a clear answer that the page never states plainly. Read those first two sentences on their own. If they need paragraph four to make sense, move the answer forward. Keep the qualification that changes the meaning, but stop making the reader hunt through setup for the thing they came to learn. 15. **Match schema to visible text, nothing more.** Check that `Organization`, `Product` or `Service`, and `FAQ` markup agree with what a person can read on the page. [In one test, schema alone barely moved AI Mode and ChatGPT results; in a live-fetch test, none of five assistants could use JSON-LD-only prices](https://forgeandsmith.com/blog/schema-wont-get-you-cited-by-ai-heres-what-its-good-for/). I used to sell schema as the citation lever. I was wrong. Visible, answer-first copy does that work; schema should not contradict it. 16. **List the page in the sitemap with an honest `lastmod`.** Confirm the exact URL appears in `sitemap.xml` and that its date reflects the last real edit. After template changes, [resubmit the updated sitemap to Google Search Console and Bing Webmaster Tools](http://replatformradar.com/blog/how-ai-answer-engines-rediscover-a-site-after-a-replatform); Bing feeds ChatGPT and Copilot. Do not stamp every URL with today’s date every day. That turns a useful change signal into noise while leaving the underlying page exactly as it was. 17. **Link to the page from somewhere crawlers already reach.** Give a money page at least one internal link from navigation, a hub, or the homepage. Check that the link exists in fetchable HTML, not only in an interaction a browser builds later. Orphaned pages can stall re-crawl for weeks to months after a replatform. That is how a carefully rewritten new page sits unseen while an older, linked page keeps getting quoted. Fix the route to the page before polishing its opening sentence. ## Entity and trust: make the business recognizable 18. **Match the business name, address, and phone everywhere you control.** Compare the site footer, contact page, and Google Business Profile character for character. Old suite numbers and abbreviations are common misses. A cleanly fetched page is less useful when the business behind it appears under conflicting details. Settle the correct information with the person who owns it, then update the public surfaces together. A crawler can report the mismatch; it cannot decide which address is real. 19. **Keep service scope and starting prices consistent.** Compare the site’s service list and prices with the profiles you control. If the page says one thing and a directory says another, an assistant has conflicting material to work with. Do not ask a tool to invent the correct price or quietly choose the newest-looking entry. Have an owner confirm what the business actually offers, update the visible copy, and then make the other controlled profiles match it. ## Monitoring: catch the next break before you rewrite 20. **Track verified AI bot fetches for each money page.** Check weekly, with at least one verified fetch per month from the bots you care about as the pass condition. [Filter logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, and PerplexityBot/User, then check IPs against vendor JSON](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). A domain-wide count can hide a service page nobody fetched. A user-agent string alone cannot tell you that the request came from the crawler named in it. 21. **Remove spoofed hits before reporting bot activity.** [Allowlist only verified IPs and signatures; user-agent strings are spoofable](https://medium.com/ai-visibility-studio/your-robots-txt-allows-ai-crawlers-your-firewall-may-still-block-them-465625e5a055). Drop any hit that fails the vendor check before it enters a dashboard or a slide. One demo log had three spoofed hits among 15. If you count those as real visits, you can declare the pipe open while the crawler you wanted has never reached the page. Verification belongs in the count, not in a footnote beneath it. 22. **Retest after every template-changing deploy.** Recheck the edge response, raw HTML, final status, and redirect path within 24 hours when a deploy changes layout, JavaScript bundling, redirects, or robots rules. A monthly review leaves room for a bad deploy to sit while the content team rewrites copy. Put the repeatable checks on a schedule and give a person ownership of the failures. This is monitoring work, not a fresh strategy workshop every time a template moves. 23. **Compare citations with verified fetches.** Record which page was fetched on which date and whether it was later cited for the prompt you track. When citations drop but fetches hold, revisit structure and trust; when both drop, start with access and retrieval. That split tells you which checks to rerun. I use [groas](https://groas.com/) for this division of labor: its SEO and AI Search work runs technical crawl and fetch checks continuously and logs fixes with reasoning, while a named strategist owns the judgment calls. For the sequence after this pre-flight, see [how to get cited in ChatGPT and Perplexity](https://groas.com/post/how-to-get-your-business-cited-in-chatgp-2). ## Final gate: check the edge before anything else 24. **Verify that the CDN and WAF admit the AI crawlers you want.** Do this first, even though it sits last on the list. Fetch a money page from outside your network using the OAI-SearchBot, Claude-SearchBot, and PerplexityBot user agents, and read both the status and the returned page. A 403, 429, or challenge shell fails the test; confirm real crawler access against verified logs, since a simulated user agent is not a verified bot. [A WAF rule wins over robots.txt](https://medium.com/ai-visibility-studio/your-robots-txt-allows-ai-crawlers-your-firewall-may-still-block-them-465625e5a055). This is the check teams skip: content sees an allow rule, engineering sees a firewall doing its job, and nobody checks the response. Skip it and the cost is every downstream improvement you paid for. The assistant cannot quote a page its crawler cannot fetch. ## Frequently Asked Questions ### Why is my page still missing from ChatGPT search even though I blocked GPTBot in robots.txt? Blocking GPTBot does not block ChatGPT search, because training bots and search bots are separate agents. Check yourdomain.com/robots.txt for a Disallow rule that covers the specific money-page URLs you want cited, and make sure OAI-SearchBot is allowed on those pages. ### Do AI crawlers execute JavaScript when they fetch a page? No. In one 41-day test, crawlers fetched JavaScript files but did not execute them and discovered no pages behind JS-only links. Ship answer pages with SSR or SSG so the fetched HTML already contains the answer copy, price, and H1 instead of an empty div shell. ### What is a soft 404 and why should I check for it? A soft 404 is an error template that still returns a 200 status code, so the URL looks fine if you only read the response header. Request the live URL after every template push and verify it returns the intended copy with a real 200, not just a passing status code. ### Where should the actual answer appear under a heading so AI assistants can quote it? Put the answer, including any number or definition, in the first two sentences under each H2, then add conditions and detail afterwards. Read those two sentences on their own; if they only make sense after paragraph four, move the answer forward. ### Can I get cited by ChatGPT or AI Mode just by adding schema markup? No. In one test, schema alone barely moved AI Mode and ChatGPT results, and in a live-fetch test none of five assistants could use JSON-LD-only prices. Visible, answer-first copy does the citation work, and schema should match that visible text, nothing more. ### How do redirect chains affect AI crawlers rediscovering my site? Chains and orphaned URLs can stall AI re-crawl for weeks, while single-hop 301s resubmitted with the sitemap recover fastest. Follow an old URL to its replacement and cut the chain to one 301 hop, then resubmit the updated sitemap to Google Search Console and Bing Webmaster Tools. ### My robots.txt allows AI crawlers, so why can a firewall still block them? A WAF rule wins over robots.txt, so the CDN or firewall can return a 403, 429, or challenge shell even when robots.txt allows the bot. Fetch a money page from outside your network using the OAI-SearchBot, Claude-SearchBot, and PerplexityBot user agents, then confirm real access against verified logs, since a simulated user agent is not a verified bot. ### How can I tell if AI crawler visits in my logs are real? User-agent strings are spoofable, so filter logs for bots like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, then check the IPs against the vendors' JSON files. Allowlist only verified IPs and signatures and drop any hit that fails the vendor check before it enters a dashboard. ## Pay For Results, Not For Hours Businesses buy the outcome, agencies resell it, and groas answers for it either way. [See If You Qualify](https://groas.typeform.com/to/xC1bQNUT) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#webpage","url":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks","name":"The AI Crawlability Pre-Flight: 24 Checks Before You Rewrite a Page","description":"Run these 24 checks for crawler access, usable HTML, answer structure and monitoring before you blame your content for missing AI citations.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/024727cf-c2f6-4eec-bf96-55f9a446de26.png"},"breadcrumb":{"@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#breadcrumb"},"dateModified":"2026-10-20T05:38:35.216Z","mainEntity":{"@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"The AI Crawlability Pre-Flight: 24 Checks Before You Rewrite a Page","item":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#article","url":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks","mainEntityOfPage":{"@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"The AI Crawlability Pre-Flight: 24 Checks Before You Rewrite a Page","description":"Run these 24 checks for crawler access, usable HTML, answer structure and monitoring before you blame your content for missing AI citations.","datePublished":"2026-10-20T05:38:35.152Z","dateModified":"2026-10-20T05:38:35.216Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/024727cf-c2f6-4eec-bf96-55f9a446de26.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"page, before, answer, check, copy, checks, url, crawlability pre-flight","wordCount":2089,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/the-ai-crawlability-pre-flight-24-checks#faq","mainEntity":[{"@type":"Question","name":"Why is my page still missing from ChatGPT search even though I blocked GPTBot in robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Blocking GPTBot does not block ChatGPT search, because training bots and search bots are separate agents. Check yourdomain.com/robots.txt for a Disallow rule that covers the specific money-page URLs you want cited, and make sure OAI-SearchBot is allowed on those pages."}},{"@type":"Question","name":"Do AI crawlers execute JavaScript when they fetch a page?","acceptedAnswer":{"@type":"Answer","text":"No. In one 41-day test, crawlers fetched JavaScript files but did not execute them and discovered no pages behind JS-only links. Ship answer pages with SSR or SSG so the fetched HTML already contains the answer copy, price, and H1 instead of an empty div shell."}},{"@type":"Question","name":"What is a soft 404 and why should I check for it?","acceptedAnswer":{"@type":"Answer","text":"A soft 404 is an error template that still returns a 200 status code, so the URL looks fine if you only read the response header. Request the live URL after every template push and verify it returns the intended copy with a real 200, not just a passing status code."}},{"@type":"Question","name":"Where should the actual answer appear under a heading so AI assistants can quote it?","acceptedAnswer":{"@type":"Answer","text":"Put the answer, including any number or definition, in the first two sentences under each H2, then add conditions and detail afterwards. Read those two sentences on their own; if they only make sense after paragraph four, move the answer forward."}},{"@type":"Question","name":"Can I get cited by ChatGPT or AI Mode just by adding schema markup?","acceptedAnswer":{"@type":"Answer","text":"No. In one test, schema alone barely moved AI Mode and ChatGPT results, and in a live-fetch test none of five assistants could use JSON-LD-only prices. Visible, answer-first copy does the citation work, and schema should match that visible text, nothing more."}},{"@type":"Question","name":"How do redirect chains affect AI crawlers rediscovering my site?","acceptedAnswer":{"@type":"Answer","text":"Chains and orphaned URLs can stall AI re-crawl for weeks, while single-hop 301s resubmitted with the sitemap recover fastest. Follow an old URL to its replacement and cut the chain to one 301 hop, then resubmit the updated sitemap to Google Search Console and Bing Webmaster Tools."}},{"@type":"Question","name":"My robots.txt allows AI crawlers, so why can a firewall still block them?","acceptedAnswer":{"@type":"Answer","text":"A WAF rule wins over robots.txt, so the CDN or firewall can return a 403, 429, or challenge shell even when robots.txt allows the bot. Fetch a money page from outside your network using the OAI-SearchBot, Claude-SearchBot, and PerplexityBot user agents, then confirm real access against verified logs, since a simulated user agent is not a verified bot."}},{"@type":"Question","name":"How can I tell if AI crawler visits in my logs are real?","acceptedAnswer":{"@type":"Answer","text":"User-agent strings are spoofable, so filter logs for bots like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, then check the IPs against the vendors' JSON files. Allowlist only verified IPs and signatures and drop any hit that fails the vendor check before it enters a dashboard."}}]}]} ```