--- title: "Before You Blame Your Content: 22 AI-Readability Checks" description: "Use this 22-point checklist to check bot access, page discovery, raw HTML, structure and verified retrieval traffic before rewriting content for AI search." url: "https://groas.com/post/the-ai-readability-pre-flight-checklist" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e3ec3198-2b39-44d9-a676-46004a679ff2.png" published: "2026-10-20T05:46:58.177Z" modified: "2026-10-20T05:46:58.245Z" --- October 20, 2026 · 8 min read # Before You Blame Your Content: 22 AI-Readability Checks [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![Cartoon: a team obsessively polishes a manuscript indoors, unaware a courier outside is blocked by a padlocked gate and a bouncer.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e3ec3198-2b39-44d9-a676-46004a679ff2.png) In this article 1. [Before you start: open the four things that show what bots receive](#before-you-start-open-the-four-things-that-show-what-bots-receive) 2. [Access: let the right agents through](#access-let-the-right-agents-through) 3. [Discoverability: give fetchers a short route to each answer](#discoverability-give-fetchers-a-short-route-to-each-answer) 4. [Renderability: make the answer visible before JavaScript runs](#renderability-make-the-answer-visible-before-javascript-runs) 5. [Structure: help a parser find the claim it came for](#structure-help-a-parser-find-the-claim-it-came-for) 6. [Verification: count real retrieval, then act on what failed](#verification-count-real-retrieval-then-act-on-what-failed) If an AI answer engine cannot get through your front gate, your better-written page will not help. Most businesses that say they are not being cited should check technical delivery before commissioning another content rewrite: a blocked user agent, a JavaScript-only answer, or a firewall challenge can stop retrieval before the engine sees a sentence. On our own site at [groas](https://groas.com/), Anthropic’s live-answer fetcher, `Claude-User`, requested `/robots.txt` 1,231 times in one 30-day window. I would start there. **Get the front gate right, then check what a fetcher can actually read.** ## Before you start: open the four things that show what bots receive - **A terminal:** Use `curl -IL` to inspect headers and redirects, and `curl -s` to inspect the returned HTML without a browser rendering it for you. - **CDN or WAF security logs:** Keep the relevant Cloudflare, Fastly, CloudFront, or AWS WAF events open so you can see challenges, blocks and rate limits. - **Your live `/robots.txt` and XML sitemap:** Open the actual URLs, not copies in a CMS or a document someone last updated eighteen months ago. - **Raw page source:** Use `view-source:` rather than the post-hydration Elements inspector. A browser can make an empty server response look complete. ## Access: let the right agents through 1. **Allow OpenAI’s search and live-answer agents separately from its training crawler.** Pass: `/robots.txt` does not block `OAI-SearchBot` or `ChatGPT-User` from the public pages you want retrieved. Fail: A blanket `Disallow: /` catches them along with `GPTBot`. [OpenAI lists its crawler tokens separately](https://developers.openai.com/api/docs/bots); decide which uses you want to permit rather than treating every OpenAI request as the same kind of visit. 2. **Check Anthropic’s agents separately, too.** Pass: `Claude-SearchBot` and `Claude-User` can reach public answer pages. Fail: A broad Anthropic block catches live retrieval while you intend only to restrict `ClaudeBot`. [The distinction between Claude’s bot tokens](https://better-robots.com/blog/claude-user-vs-claudebot-vs-claude-searchbot) matters here. If `Claude-User` cannot request a page, polishing that page will not make it available for a live fetch. 3. **Check `PerplexityBot` against every applicable robots rule.** Pass: Your live `/robots.txt` permits access to the public pages you want it to find. Fail: A restrictive generic rule blocks it while the team assumes all search bots have an exception. Read the whole file, including rules added for older anti-scraping projects; the agent name in a planning document does not override the response served by your site. 4. **Keep `Googlebot` distinct from `Google-Extended`.** Pass: `Googlebot` can reach the public pages and resources you want in Google Search; any `Google-Extended` restriction reflects a deliberate separate choice. Fail: Blocking `Googlebot` while trying to manage AI visibility. Do not use the `Google-Extended` control as a substitute for checking whether Google can crawl and index the page in the first place. 5. **Serve the answer without a login, registration screen or hard consent gate.** Pass: An unauthenticated request gets an HTTP 200 response containing the primary informational text. Fail: It gets a 401, a 403, or HTML consisting mainly of a sign-in or consent screen. Check the raw response, not the page you see after accepting cookies and signing in. A fetcher cannot cite an answer it was never served. 6. **Review datacenter blocks without opening the door to every datacenter.** Pass: Your security rules distinguish verified agents you want to admit from generic cloud-hosted traffic. Fail: A blanket commercial-cloud block catches legitimate retrieval requests. The fix is a scoped, verified exception, not unrestricted access for entire AWS, Azure or Google Cloud IP ranges. Check the relevant WAF events before changing a rule; an absent analytics visit is not proof that no request arrived. 7. **Give legitimate retrieval bursts room to complete.** Pass: Verified agent requests can fetch several relevant URLs without immediately hitting an edge rate limit. Fail: A tight per-IP cap, such as 10 requests a minute, returns HTTP 429 partway through a multi-page visit. Inspect the response codes and timing in your logs. A request that reaches the edge but never receives page HTML has not passed this check. ## Discoverability: give fetchers a short route to each answer 8. **Declare a current XML sitemap in `/robots.txt`.** Pass: A `Sitemap: https://yourdomain.com/sitemap.xml` directive points to a usable sitemap containing canonical, 200 OK URLs. Fail: The directive is missing, the file is stale, or its listed URLs redirect. Check the live file after publishing new pages. A sitemap is not a replacement for links, but it should not send a fetcher through broken or obsolete addresses. 9. **Do not make `/llms.txt` carry a broken site architecture.** Pass: Priority pages are discoverable through ordinary links and a working sitemap whether or not you publish that file. Fail: The team writes an experimental root file while answer pages remain orphaned. Keep the hierarchy of work straight: first make the URLs reachable, then decide whether another discovery file is worth maintaining. 10. **Link priority answer pages from static HTML within three hops of the homepage.** Pass: Documentation, pricing and comparison pages are reachable through ordinary `` links. Fail: A page exists only behind infinite scroll, a click listener or an internal search result. If you cannot follow the route by reading returned HTML and opening its links, do not assume a non-rendering fetcher will find it. 11. **Point internal links straight to the final URL.** Pass: A link to a canonical answer page returns 200 OK without a detour. Fail: It walks through old HTTP-to-HTTPS, host-change or 301-to-302 chains. Run `curl -IL` on the actual internal link, not merely on the destination you think it uses. A redirect may work in your browser while adding an avoidable step to automated retrieval. ## Renderability: make the answer visible before JavaScript runs ![A browser shows interactive tabs while a raw HTML fetch shows an empty page shell.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e8420fde-0d28-4bf1-ae4a-4288c015c853.png) 12. **Put primary claims in the initial HTML response.** Pass: Definitions, specifications, prices and direct answers appear in the HTML returned before scripts execute. Fail: The response is an empty `
` shell whose content arrives later through client-side JavaScript. Test the response with `curl -s` or `view-source:`. Do not use the finished browser page as evidence that a fetcher received its text; those are different views of the same URL. 13. **Keep accordion and tab answers in the initial HTML.** Pass: The text is present before a click and its visibility changes through CSS or native `
` and `` elements. Fail: An interaction mounts the answer for the first time. I have seen plenty of pages that look complete to a visitor and empty to a raw fetch. The check is simple: search the returned source for a distinctive sentence inside the closed panel. 14. **Measure response time for the requests that matter.** Pass: Your origin or edge returns answer-page HTML promptly; the draft target is a Time to First Byte below 600 milliseconds. Fail: Uncached execution regularly takes more than 1,500 milliseconds before sending a first byte. Inspect actual request timing rather than relying on a browser test from one location. A live-answer fetch that stalls has less opportunity to contribute its page to the response. ## Structure: help a parser find the claim it came for 15. **Answer the page’s main question near the top.** Pass: The direct answer, relevant price range or core specification appears under the main heading within roughly the first 150 words. Fail: A visitor must pass several blocks of generic background before reaching it. This is a page-composition check, not permission to invent a neat answer. If your numbers are real but buried, use the [AI Overview landing page pre-flight](https://groas.com/post/the-ai-overview-landing-page-pre-flight-2) to check where they appear. 16. **Use semantic headings rather than styled generic containers.** Pass: An `

` names the page and clear `

` and `

` headings lead into their answers. Fail: CSS makes a `
` look like a heading without giving the HTML that structure. Inspect the markup with styling removed. A readable hierarchy helps both people skimming the page and systems trying to separate one answer from the next. 17. **Make structured data agree with visible copy.** Pass: Any `FAQPage`, `Product` or `Organization` JSON-LD reflects what the page actually says. Fail: Schema retains an old price, lists a question the page does not answer, or makes a claim that visitors cannot find. Compare the rendered copy and the structured data after substantive edits. Markup cannot repair a contradiction between two versions of the same page. 18. **Use native HTML tables for genuinely tabular information.** Pass: Specifications, fees or feature comparisons use ``, rows and cells that preserve their relationships in the source. Fail: Nested grid `
` elements depend on CSS positioning to show which value belongs to which label. Check a text-only view before approving the layout. If the relationships collapse when styling disappears, the comparison needs better markup. 19. **Resolve canonical URLs and their responses together.** Pass: A clean 200 OK answer page declares the intended canonical URL, and that URL itself resolves directly. Fail: The tag points to a redirect, a broken path or an unintended variation. Check protocol, host and trailing slash against the served URL. Do not ask a retriever to work out which of several conflicting addresses deserves the attribution. ## Verification: count real retrieval, then act on what failed 20. **Verify bot identity before reporting AI traffic.** Pass: Where the vendor provides a verification method, logs match requests against its published IP ranges or use forward-confirmed reverse DNS. Fail: A dashboard counts any request whose `User-Agent` claims to be `Claude-User` or `ChatGPT-User`. That header alone is not proof of identity. Without verification, a reassuring traffic chart can conceal the fact that the agents you care about are being blocked. ![A server checkpoint verifies an AI retrieval agent while a spoofed user agent waits outside.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/7dad25d5-edd1-429c-b1cc-927e6675411e.png) 21. **Separate live-answer fetches from training crawls, and assign fixes someone can execute.** Pass: Logs distinguish tokens such as `Claude-User` and `ChatGPT-User` from `ClaudeBot` and `GPTBot`, show their response codes on priority URLs, and give each failure an owner. Fail: A single “AI bot traffic” total is treated as evidence of citations. Our [30-day crawler-log teardown](https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw) shows why the distinction is useful. An audit dashboard can flag a blocked request; it cannot change your WAF rule. [groas for businesses](https://groas.com/for-businesses) takes an execution-first approach to search work, but edge and site changes still need the right access and ownership. 22. **Check the WAF challenge before you commission more content.** Pass: Verified agents you intend to admit receive page HTML rather than a managed challenge, JavaScript puzzle or HTTP 403. Fail: Bot Fight Mode or a similar edge rule challenges automated traffic indiscriminately. Use the [Cloudflare bot troubleshooting guide](https://developers.cloudflare.com/bots/troubleshooting/false-positives/) and your own security logs to inspect the actual response, then scope any exception to verified traffic. This is the check teams skip most: they pay for thought leadership while the firewall turns away the fetcher before it reads a word. **The cost is content the intended agent cannot reach.** ## Frequently Asked Questions ### What should I check before assuming content isn't being cited by AI search engines? Businesses should first check technical delivery methods such as blocked user agents, JavaScript-only answers, or firewall challenges. Polishing content will not help if a retrieval engine cannot access the page through the front gate. Examine CDN or WAF security logs and the live robots.txt file to diagnose blockages. ### Why is it important to differentiate between live-answer fetches and training crawls in log analysis? Distinguishing between different tokens is crucial because a single 'AI bot traffic' total does not confirm citations. Logs must show response codes for priority URLs and assign ownership to each failure. This distinction allows for targeted fixes rather than assuming traffic is successful. ### How can I ensure AI fetchers can find priority answer pages? Priority answer pages should be reachable through ordinary HTML links within three hops of the homepage. If a page only exists behind infinite scroll or a click listener, a non-rendering fetcher likely will not find it. Check the returned HTML to verify link accessibility. ### How do I make sure my content is visible before JavaScript runs? Definitions, specifications, and direct answers must appear in the raw HTML returned before any scripts execute. Testing the response with curl -s confirms if a fetcher received the text. Do not use the finished browser page as evidence of what a fetcher received. ### What structure should I use to help AI parsers understand the content? Use semantic headings such as H1, H2, and H3 rather than styled generic containers. A clear hierarchy helps both human readers and AI systems separate individual answers. Additionally, use native HTML tables for genuinely tabular information. ### How can I verify if AI traffic is legitimate? Without verification, relying on a dashboard that counts any request claiming a specific User-Agent is misleading. Verification requires logs to match requests against the vendor's published IP ranges or use forward-confirmed reverse DNS. This confirms that the agent is who it claims to be. ## Pay For Results, Not For Hours Businesses buy the outcome, agencies resell it, and groas answers for it either way. [See If You Qualify](https://groas.typeform.com/to/xC1bQNUT) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#webpage","url":"https://groas.com/post/the-ai-readability-pre-flight-checklist","name":"Before You Blame Your Content: 22 AI-Readability Checks","description":"Use this 22-point checklist to check bot access, page discovery, raw HTML, structure and verified retrieval traffic before rewriting content for AI search.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e3ec3198-2b39-44d9-a676-46004a679ff2.png"},"breadcrumb":{"@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#breadcrumb"},"dateModified":"2026-10-20T05:46:58.245Z","mainEntity":{"@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Before You Blame Your Content: 22 AI-Readability Checks","item":"https://groas.com/post/the-ai-readability-pre-flight-checklist"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#article","url":"https://groas.com/post/the-ai-readability-pre-flight-checklist","mainEntityOfPage":{"@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Before You Blame Your Content: 22 AI-Readability Checks","description":"Use this 22-point checklist to check bot access, page discovery, raw HTML, structure and verified retrieval traffic before rewriting content for AI search.","datePublished":"2026-10-20T05:46:58.177Z","dateModified":"2026-10-20T05:46:58.245Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/e3ec3198-2b39-44d9-a676-46004a679ff2.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"pass, before, page, fail, check, answer, html, content","wordCount":1764,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/the-ai-readability-pre-flight-checklist#faq","mainEntity":[{"@type":"Question","name":"What should I check before assuming content isn't being cited by AI search engines?","acceptedAnswer":{"@type":"Answer","text":"Businesses should first check technical delivery methods such as blocked user agents, JavaScript-only answers, or firewall challenges. Polishing content will not help if a retrieval engine cannot access the page through the front gate. Examine CDN or WAF security logs and the live robots.txt file to diagnose blockages."}},{"@type":"Question","name":"Why is it important to differentiate between live-answer fetches and training crawls in log analysis?","acceptedAnswer":{"@type":"Answer","text":"Distinguishing between different tokens is crucial because a single 'AI bot traffic' total does not confirm citations. Logs must show response codes for priority URLs and assign ownership to each failure. This distinction allows for targeted fixes rather than assuming traffic is successful."}},{"@type":"Question","name":"How can I ensure AI fetchers can find priority answer pages?","acceptedAnswer":{"@type":"Answer","text":"Priority answer pages should be reachable through ordinary HTML links within three hops of the homepage. If a page only exists behind infinite scroll or a click listener, a non-rendering fetcher likely will not find it. Check the returned HTML to verify link accessibility."}},{"@type":"Question","name":"How do I make sure my content is visible before JavaScript runs?","acceptedAnswer":{"@type":"Answer","text":"Definitions, specifications, and direct answers must appear in the raw HTML returned before any scripts execute. Testing the response with curl -s confirms if a fetcher received the text. Do not use the finished browser page as evidence of what a fetcher received."}},{"@type":"Question","name":"What structure should I use to help AI parsers understand the content?","acceptedAnswer":{"@type":"Answer","text":"Use semantic headings such as H1, H2, and H3 rather than styled generic containers. A clear hierarchy helps both human readers and AI systems separate individual answers. Additionally, use native HTML tables for genuinely tabular information."}},{"@type":"Question","name":"How can I verify if AI traffic is legitimate?","acceptedAnswer":{"@type":"Answer","text":"Without verification, relying on a dashboard that counts any request claiming a specific User-Agent is misleading. Verification requires logs to match requests against the vendor's published IP ranges or use forward-confirmed reverse DNS. This confirms that the agent is who it claims to be."}}]}]} ```