Postmortem: The Site Change That Makes Your Pages Disappear from AI Answers

A page can look fine in your browser and disappear from AI answers the same day. Most lost AI visibility is not a content problem. It is a plumbing problem: a routine site change refuses the bots that fetch your page live, or gives them an empty shell. The assistant cites a page it can read instead. Your rankings may still look normal.

Our edge logs made that mechanism hard to shrug off. Over a 30-day window, robots.txt drew 3,197 verified bot hits. Claude-User accounted for 1,177 of them during live-answer fetching. That is a lot of gate checks for something many teams treat as a file they only revisit during a migration. When the gate says no, or the fetch behind it fails, there is no helpful alert saying your next citation went elsewhere.

Live fetchers like ChatGPT-User, Claude-User and Perplexity-User pull pages when a user asks a question. If they cannot get the page, the assistant has to work with what it can reach. A CDN setting, a robots.txt rewrite, a JavaScript-heavy redesign or a URL swap without a working redirect can all create that gap. The human visitor walks through; the bot stops at the door.

The setup: a change nobody treats as a content change

Take the version of this every team eventually runs into. This is a typical composite, not a report of one client incident. A marketing site gets a routine infrastructure or design update. Nobody changes the blog copy or decides to withdraw the company from AI answers. The site loads for employees. Rankings hold. Search Console looks calm. Weeks later, someone asks an assistant to recommend a vendor in the category and the brand is missing.

The change could be a CDN control meant to reduce scraper traffic. It could be a template that moves the useful text into JavaScript-rendered tabs. It could be a migration that sends an old cited URL somewhere unexpected. Those are different changes with the same practical consequence: the version a person sees is not necessarily the version an answer bot gets.

That distinction matters more now because there is not one universal “AI bot” setting. Training crawlers such as GPTBot and Google-Extended, search and answer bots such as OAI-SearchBot, and live fetchers such as Claude-User do different jobs. You may choose to block a training crawler. That decision should not silently block the agents you want to find, fetch and cite your pages. Group them all under “AI scraping” and you can make the wrong rule look tidy in a dashboard.

Cloudflare began blocking AI crawlers by default on new domains unless owners grant permission. That makes the control worth checking, but the label in a dashboard is not the answer. What matters in this postmortem is whether the agents involved in search and live answers can actually fetch the pages you want cited. Test the response, not the setting’s name.

The first sign: the page is available to people, not to the fetcher

The earliest useful warning is not a fall in rankings. It is a mismatch between two requests to the same URL. A person gets the page; an AI user agent gets a refusal. Or both get a 200 response, but the bot receives HTML without the passage that would answer the question.

The refusal can happen at the edge. A robots.txt checker may report no block while the server still refuses the request. After Cloudflare retired its Managed robots.txt, researchers found sites where the visible robots block had gone but edge enforcement remained. A clean-looking file does not prove that the request gets through. Neither does opening the page in Chrome.

The empty-shell version is quieter. The OpenAI, Anthropic and Perplexity agents in one test read the raw first-response HTML rather than executing JavaScript. If the relevant pricing table, review or product detail arrives only after a script runs, that tested fetcher does not get it. View source and search for a distinctive sentence from the page. If it is missing, the attractive browser rendering is beside the point.

Our robots.txt counts explain why I take the first mismatch seriously. Claude-User checked that gate 1,177 times in 30 days during live-answer fetching. The gate is part of the answer path, not just a housekeeping stop on a distant crawl schedule. A bad rule can affect the next relevant fetch; you do not get to assume there will be a leisurely recrawl before it matters.

The wrong call: rewrite the copy while the door stays shut

The team sees a missing citation and reaches for the tool it knows: content. It refreshes headings, rewrites a comparison page and asks whether the answer passage is direct enough. Those may be sensible edits when a fetcher can read the page. They cannot repair a 403, and they cannot make text in a JavaScript-only tab appear in raw HTML.

I understand the instinct. In paid search, a landing page can look acceptable to the person who approved it while the system interacting with it gets a worse page. A client once swapped a landing page without telling me; the Quality Score fell, and the investigation led back to what had changed on the page, not to the ads. The useful habit from that experience is to inspect the thing the system receives before polishing the message you wish it received.

Here, the wrong call lasts because the familiar diagnostics stay reassuring. A page can remain indexed while a redesign changes the machine-readable content, headings, schema or redirects that support AI visibility. Search Console is useful, but a steady search report is not a live-fetch test. It does not settle whether Claude-User can read today’s pricing passage or whether an old cited URL reaches its intended replacement.

So the content team does another pass. The copy improves. The blocked request does not. Before rewriting a word for a lost citation, check what the fetcher gets.

The damage: another page supplies the answer

The assistant does not pause to explain that your CDN refused its request. It answers with accessible material. For this composite, that means a competitor with a readable page and a working path to it takes the citation. You do not merely rank a little lower in that answer; your page is absent from the material the assistant could use.

One account of blocked Perplexity search access describes citations falling to near zero within about two days. The exact timetable is not a promise for every site or every assistant. The mechanism is the point: a live fetch that fails at answer time can cost that answer immediately, while other visibility measures remain slow to tell you what happened.

Then the team waits for a dashboard to confirm the problem. But AI visibility trackers generally re-run selected prompts and log mentions across assistants. They cannot tell you about every question a buyer asks. Citations can also change without a corresponding change on your page, so the first missing mention is easy to dismiss as noise. In the composite, that is how a routine deploy turns into weeks of looking in the wrong place.

The cost is especially awkward when you also pay to win search traffic. Say you spend $20k a month on paid search while an answer above the ad names someone else. The paid campaign and the AI citation are different paths, but the question behind them may be the same buying decision. Spending more attention on ad copy will not reopen a refused page. Prove the fetch works before you diagnose the answer.

The fix: make the live fetch part of release QA

The CDN admin was trying to stop unwanted traffic. The designer was trying to ship a better page. The content team was trying to recover a mention. None of those jobs is the mistake. The mistake is treating access for search and live-answer bots as nobody’s release criterion.

Paid landing pages usually get checked before launch because a broken page can waste spend as soon as traffic arrives. A page you want cited deserves the same practical check: request it as the relevant agent, inspect the returned HTML, follow the old URL if it changed, then see what the answer surfaces do after release. That sequence separates a blocked door from a weak passage. It also gives the team somewhere useful to start when a mention disappears.

I would do this before the change merges and again on the day it ships. A pre-flight catches the planned mistake; a post-flight catches the rule, cache or redirect that behaved differently in production. The checklist is short on purpose. This is not an invitation to turn every deploy into a conference about “AI readiness.” It is a request to verify that the page can be read.

The release checklist that catches the quiet failure

Run these checks on the pages you most need an assistant to cite. Keep the pre-change responses so you can compare them with production. For the allow-rule details, use the AI crawler swipe file; for a fuller request-level check, use the curl and CDN audit.

Robots and edge

1. Check CDN AI-crawler controls for search and live-answer access. A dashboard label alone does not prove which agents can pass.
2. Fetch robots.txt and a priority page as OAI-SearchBot, PerplexityBot and Claude-User. Compare their responses with an ordinary browser request; a browser 200 does not rule out a bot 403.
3. Diff robots.txt before and after the deploy. Investigate any new Disallow affecting an agent you intend to allow.

HTML and page structure

4. View source and find a distinctive sentence from the answer passage. A sentence visible only after JavaScript runs was not in the raw HTML those tested agents read.
5. Keep important pricing, specifications, reviews and answer passages in first-response HTML rather than relying on tabs or load-more controls.
6. Check that a redesign has not flattened useful H2/H3 structure or dropped existing schema.

URLs and answers

7. Fetch changed URLs and their redirects as a bot, not just in your browser. A redirect that works for you does not prove the bot reaches the replacement page.
8. Check canonicals on migrated priority pages against the intended destination.
9. Re-run three buying prompts in ChatGPT, Perplexity and AI Overviews on ship day, and save the answers. This is the check teams most often skip. Without a same-day comparison, a missing citation can spend weeks being called noise while another page supplies the answer. If the spot-check exposes a fetch problem, work through the technical fixes for AI visibility before rewriting copy.

The rule after the postmortem

If AI answers drive none of your pipeline, this may not be your release priority. If buyers ask an assistant what to buy in your category, it is. You cannot infer bot access from a healthy-looking page, a stable ranking or a reassuring robots.txt checker. Each tells you something narrower than the thing you need to know.

The rule I apply now is simple: no site change is done until the live fetch can reach and read the page you want cited.