AI Visibility Starts With Access, Not Schema: A Technical Triage

If your brand has vanished from AI answers, the first fix probably is not another schema audit. Check whether the engines can reach and read the page at all.

I learned that order in paid search. When an ad group stopped converting overnight, I did not begin by rewriting the ads. I checked the landing page: had someone broken the checkout form, changed the URL or shipped a redirect nobody noticed? The same instinct belongs in AI visibility. A pristine FAQPage block cannot rescue an answer trapped behind a firewall challenge or loaded only after JavaScript runs.

That does not make structured data useless. It makes it sixth on the list. Here is the technical triage I would run before commissioning an eighty-item JSON-LD spreadsheet: access, readable HTML, indexation, speed, discoverability and, finally, schema. The order matters because each failure can stop the next improvement from being seen.

Why can a page work for visitors but disappear from AI answers?

A buyer asks ChatGPT Search, Claude or Perplexity about your category. The service searches or retrieves candidate pages and tries to extract useful text. If it cannot fetch yours, or receives an empty shell instead of the answer, it can use another page. You do not get a conversion alert saying, ‘Your competitor was cited instead.’

Classic search at least gives you places to investigate, including crawl and indexation reports. AI citations can change without an equally clear warning in your analytics. Human traffic may look normal while an automated fetcher receives a 403 Forbidden response or a page with no substantive text. A healthy-looking browser session is not proof of a healthy retrieval path.

Which bots am I actually trying to let in?

Training crawlers and search-related crawlers do different jobs. OpenAI’s crawler overview distinguishes GPTBot, used for training, from OAI-SearchBot, used for ChatGPT Search, and ChatGPT-User, which can fetch pages during a conversation. Anthropic’s crawler guidelines likewise distinguish ClaudeBot from Claude-SearchBot and Claude-User.

That distinction lets you make an intentional choice about training access without accidentally blocking search access. A blanket ‘block AI bots’ rule does not make that choice for you; it hides it inside a wildcard. Name the agents and decide what each may do.

Step 1: Can search crawlers get through my rules and firewall?

Start with robots.txt at the root of your site. Look for broad disallows and rules covering the pages you want cited. Blocking GPTBot is a different decision from blocking OAI-SearchBot; the same distinction applies to ClaudeBot and Claude-SearchBot. If you want a starting point for the rules, use the AI crawler swipe file, then check its examples against your actual configuration rather than pasting them blind.

Perplexity presents another pair to check. Its bot documentation identifies PerplexityBot for discovery and indexing and Perplexity-User for retrieval on behalf of users. Blocking the discovery bot while wondering why new pages do not appear is a fairly expensive way to learn that distinction.

Then inspect the layer in front of the server. An open robots.txt file does not help if a CDN or web application firewall serves the fetcher a JavaScript challenge or a 403. Cloudflare’s bot troubleshooting guidance is worth reading if its security features sit in your request path: some bot controls do not behave like ordinary WAF rules. Do not solve this by trusting any request that merely claims to be an AI bot. Verify the requester, then configure the appropriate access rule with your security team.

The pass condition is a verified search-related bot reaching the intended URL and receiving the intended page, not a developer saying that the site loads on their laptop. This is the AI-search version of checking whether the landing-page form still submits before you touch the ad copy.

Step 2: Is my answer in the HTML a fetcher receives?

Modern sites often assemble the useful part of a page in the browser. React, Next.js, Vue and Nuxt can serve readable HTML, but they can also serve a shell that depends on client-side JavaScript. Do not infer which one you have from the framework name. Check the response.

Some AI crawlers and retrieval systems do not execute client-side scripts the way a human visitor’s browser does. Nobori’s analysis of crawler rendering behavior describes that gap. If the product explanation or pricing table appears only after hydration, a fetcher reading the initial response may miss it. Put the answer you want retrieved in the initial HTML.

How do I check without buying a crawler?

Run a request against a page that should answer a buyer’s question:

curl -s -A "OAI-SearchBot" https://yourdomain.com/page | grep -i "your key answer"

Replace the URL and search phrase with your own. If the phrase is missing, inspect the full response before concluding why: you might have received a challenge, a redirect, different wording or an HTML shell. Changing the user-agent string is a diagnostic shortcut, not proof that the requester has been verified as that bot.

You can also disable JavaScript in your browser’s developer tools and reload the page. If the main answer becomes a spinner or blank space, ask for server-side rendering or static generation for that content. The goal is not to make every interactive feature work without JavaScript. It is to make the information a buyer needs readable before the scripts run.

Step 3: Am I pointing engines to the right version of the page?

Once a fetcher can read the page, check whether your site sends it somewhere else. CMS parameters, regional versions and staging environments can create several URLs for similar content. Canonical tags and redirects should identify the version you mean to publish, not point at an old campaign page or send a crawler around a chain of hops.

I would inspect the canonical tag on each priority page, follow its redirects and check the internal links that lead to it. A self-referential canonical can be appropriate for a preferred public URL; a page with legitimate variants needs an intentional canonical choice. The job is to remove contradictory signals, not to paste the same tag onto every URL.

This is the quiet landing-page swap in another costume. The page may still load for you while the version you want discovered is no longer the version your site recommends.

Step 4: Does the page arrive quickly and with the answer intact?

A live fetch has limited patience. Slow server responses, heavy documents and repeated redirects all make a page harder to retrieve promptly. I would measure the response before debating headline phrasing: check time to first byte, follow the redirect path and examine the size of the raw HTML.

Page weight matters particularly when frameworks embed large hydration data, inline SVGs, scripts or CSS ahead of the useful copy. DebugBear’s breakdown of Googlebot’s HTML limit explains how an oversized response can be truncated for Googlebot. That is a reason to inspect what arrives within the fetched document, not a license to claim every AI engine applies the same cutoff.

Keep critical answers in readable HTML near the top of the response. Strip unnecessary bulk where you can, then fetch the page again and confirm that the explanation or comparison table is still present. A fast page that omits its answer has merely failed more efficiently.

Step 5: Can a crawler find the page and extract its answer?

A readable page still needs a route in. If the only path to a useful comparison is a pagination script or a link buried several levels down, give it a direct internal link from a relevant, accessible page. Then look at the destination as a reader would: does it answer the question, or does it spend four paragraphs announcing that the brand is ‘committed to innovation’?

Use clear H2 and H3 headings that match the questions buyers ask. Put the direct answer immediately below the relevant heading, then explain the detail and tradeoffs. Make the useful passage easy to reach and easy to recognise. You do not need to turn decent prose into a wall of robotic FAQs to do that.

Step 6: Is schema worth fixing now?

Yes, when the earlier checks pass and the markup serves a clear purpose. Structured data can clarify entities and relationships and support features in conventional search. It cannot make a blocked fetch succeed or supply body text that never reaches the initial HTML.

That is why I put schema last, not in the bin. Doublebricks’ review of schema and AI citations discusses the limited evidence for treating JSON-LD alone as a citation lever. The practical decision is simpler than a fight over studies: make the page accessible and readable before polishing its labels. An agency proposing a major schema overhaul while search fetchers receive a firewall challenge has the order backwards. Nice paint. No engine.

How do I verify the fixes instead of trusting a dashboard?

Start with server and edge logs. Look for requests to the specific URLs you changed, their status codes, redirects and timing. A prompt-tracking chart can tell you what an answer displayed; it cannot, by itself, tell you whether your firewall served a challenge to a fetcher.

There is a catch: user-agent strings are easy to spoof. Screaming Frog’s guide to bot verification in logs explains why a request labelled OAI-SearchBot is not automatically an authentic visit. Cross-check IP information against official bot information or use appropriate DNS verification before treating a hit as evidence. Our walkthrough of what hits your server when ChatGPT looks for an answer goes deeper on the log-filtering process.

I would verify in this order:

1. Fetch the priority URL directly and confirm that it returns the intended page with the key answer in its initial HTML.
2. Check the edge and server logs for verified search-related bot requests, clean responses and unexpected challenges or redirects.
3. Repeat representative buyer prompts in the AI services you care about and record whether the page appears or is cited.

That last step checks the outcome; it is not a promise of a citation on a schedule. A clean 200 OK proves access, not that an engine will choose your answer. If citations still do not appear, you can investigate content and relevance without wondering whether the door was locked all along.

What should I leave off the roadmap for now?

Skip work that cannot repair the failure you just found:

• Micro-tuning schema properties while a fetcher cannot read the page. Set sensible markup later; do not spend twenty hours debating a founder-bio property before fixing access.
• Rewriting clear copy into formulaic FAQ blocks. Make the answer easy to find, but keep the page useful to the person who clicks through.
• Submitting URLs to low-quality AI directories as a substitute for fixing discovery and internal links.
• Buying another passive prompt dashboard when nobody owns the server, rendering or firewall issue behind the missing citation.

This is not an argument against measurement. It is an argument against measuring a leak indefinitely while leaving the pipe open.

When should I hand this off, and what should I demand?

Hand it off when the work crosses teams: marketing can identify the missing answers, but engineering or security may own rendering, CDN rules and deployment. Give whoever takes it the priority URLs, the failed fetches and the business questions those pages should answer. Ask for changes you can verify in the response and logs, not an eighty-row spreadsheet that waits in a backlog.

The same standard applies to a search partner. I would ask who keeps watch after the first repair, who decides which failures matter and who is accountable for qualified pipeline rather than a prettier audit score. A diagnostic tool is not an operating model. Traditional agency reporting is not one either if every actionable finding ends at a meeting.

That is the case for groas over a passive tracker or a review-and-recommend retainer. groas combines autonomous execution across paid and organic search with a named human strategist who sets direction and guardrails. Its focus is attributable revenue and qualified pipeline, not merely more charts. Server and firewall changes still need to be owned and verified by the people with access to those systems; I would not confuse an autonomous growth engine with permission to rewrite your security policy.

Today, start with one page you want cited. Fetch its initial HTML, check its bot access and follow its redirects. If the answer is missing or the request is blocked, fix that before you buy a schema checklist. If you want the broader search work run continuously rather than left as another set of recommendations, see how groas runs search work around the clock for a flat monthly fee.