---
title: "Postmortem: The Access Failure Behind a Brand’s Absence From AI Answers"
description: "When a business disappears from ChatGPT and Perplexity answers, more content may not help. Trace the failure from crawler access to the page AI can actually read."
url: "https://groas.com/post/postmortem-how-a-business-goes-invisible"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a351b684-3f64-40e5-96cd-a7e616ade69f.png"
published: "2026-10-20T05:36:47.693Z"
modified: "2026-10-20T05:36:47.762Z"
---

October 20, 2026 · 9 min read

# Postmortem: The Access Failure Behind a Brand’s Absence From AI Answers

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Miniature movers stack fine furniture outside a showroom whose door is bricked shut: good content piling up where no visitor can get in.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a351b684-3f64-40e5-96cd-a7e616ade69f.png)

In this article

1. [The setup: 40 posts and a brand nobody mentions](#the-setup-40-posts-and-a-brand-nobody-mentions)
2. [The wrong call: commission more content before checking access](#the-wrong-call-commission-more-content-before-checking-access)
3. [First break: the edge closes before the crawler reaches the site](#first-break-the-edge-closes-before-the-crawler-reaches-the-site)
4. [Second break: the crawler rules confuse training with retrieval](#second-break-the-crawler-rules-confuse-training-with-retrieval)
5. [Third break: the page returns a 200 but not the answer](#third-break-the-page-returns-a-200-but-not-the-answer)
6. [Fourth break: the readable page makes the answer hard to find](#fourth-break-the-readable-page-makes-the-answer-hard-to-find)
7. [The earlier change nobody rechecked](#the-earlier-change-nobody-rechecked)
8. [The fix: work from the edge to the answer](#the-fix-work-from-the-edge-to-the-answer)

A business can publish a year of good content and still be absent when buyers ask ChatGPT who to choose. The team blames the writing. I’d check the door first: a crawler gets blocked, rendering breaks, or a site change swaps out the page the system can read.

That order matters. On groas.com last month, `/robots.txt` was the most-fetched path on the site: **15,515 fetches**, including **1,231 live-answer fetches from Claude-User**. The first file an AI reads is the one nobody on the marketing team ever opens.

## The setup: 40 posts and a brand nobody mentions

Take an explicitly typical composite, not a claim about a particular client. A B2B company publishes 40 solid posts in a year. Its product pages are clean, pricing is public, and the blog answers buyer questions instead of circling them for 900 words. Search rankings look fine.

Then someone on the marketing team runs ten buyer prompts through ChatGPT and Perplexity. Competitors appear in six. Their own brand appears in **zero**. Not ranked low. Not described badly. Just absent.

**Absence is the first sign, not the diagnosis.** The team has evidence that the brand is missing from those answers. It does not yet have evidence that another article will fix it. Those prompts give the team a reason to investigate, not a verdict on every page it has published.

I know how tempting that leap is. When a paid-search account goes sideways, the visible thing gets blamed first: ads, bids, keywords. I’ve chased those while a landing-page change sat underneath the problem. The lesson was not that copy never matters. It was that I had to check what the system could actually read before rewriting what I could see.

## The wrong call: commission more content before checking access

In this composite, the team decides its content needs more authority. It briefs another batch of articles, expands existing posts, and spends its next review meeting debating headings. None of those decisions is foolish on its own. They are foolish **before anyone checks whether AI fetchers can reach the pages**.

The silence continues. The new posts join the old ones, and the team reads that silence as confirmation that it needs still more content. This is the damage: useful work accumulates behind an access failure, while every new brief points effort away from the cause. Worse, the team has no way to tell whether the latest post failed on its merits or was never available to the system answering the question.

Our logs show why I would start elsewhere. On groas.com, no product page or blog post came close to `/robots.txt` at **15,515 fetches** last month. **1,231 were live-answer fetches from Claude-User.** Those requests are a reminder that access is not a technical preamble somebody can safely ignore. It sits in the path between a buyer’s question and the page that could answer it.

From there, I would reconstruct the failure in order. Each stage can hide the next. There is little value debating the first paragraph of a pricing page while a CDN refuses the request.

## First break: the edge closes before the crawler reaches the site

A security setting changes. Bot protection goes on, or the site launches on a new Cloudflare domain. From [1 July 2025, those domains block GPTBot, ClaudeBot and PerplexityBot by default](https://seoscanpro.ai/blog/how-to-rank-in-chatgpt-search/). The site still works for human visitors and ranks in Google, so the content team has no obvious reason to suspect the door closed.

The edge logs would have told a different story. In [one test of 314 live sites, 43% refused at least one AI crawler, but only 23 had a written robots.txt rule](https://www.lovedby.ai/blog/blocking-ai-crawlers-study). The other 112 refused at the server with nothing written down. Reading the site’s robots file alone would not have caught those refusals.

That is the first check I would make in the composite: **are the relevant requests getting through the edge, or getting blocked before page rules matter?** I would look at the requests and responses, not just whether the site loads on my laptop. If the CDN says no, publishing another article changes nothing.

![A locked filing cabinet labeled robots.txt blocking a robot holding a quote card](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d11ef415-6032-4bd5-ba29-071dfeceb257.png)

## Second break: the crawler rules confuse training with retrieval

Suppose the edge allows requests. The next person checks `/robots.txt` and finds a rule copied from a conversation about AI training. Someone wanted to stop `GPTBot` and assumed that meant stopping ChatGPT citations, or allowed one bot and assumed that covered every route into answers.

Those are different jobs. [OAI-SearchBot builds the ChatGPT search index, while GPTBot collects training data](https://www.lifewood.ai/ai-crawlers.html). A rule for the training crawler does not stand in for a rule for the search crawler. If the intention is to block training while permitting search discovery, give those agents **separate, deliberate groups** rather than copying one blanket instruction and hoping it means both. The [copy-paste robots.txt rules](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob) show the split.

The owner may not even have written the rule. In one crawl study, [18 of 23 sites with robots.txt AI rules had them injected by Cloudflare Managed Content](https://www.madx.digital/learn/ai-crawlers). A marketing team can spend weeks rewriting pages under a policy it never knew had shipped.

Robots rules also do not tell the whole live-fetch story. [PerplexityBot builds an index and respects robots.txt, while Perplexity-User fetches live for a question and generally ignores it](https://www.okara.ai/blog/how-to-get-cited-by-perplexity). [ChatGPT-User likewise makes user-initiated fetches](https://www.madx.digital/learn/ai-crawlers). So I would check each relevant agent and its response, not announce that one robots change has restored every kind of visibility. **Separate indexing, training, and live access before calling the problem solved.**

## Third break: the page returns a 200 but not the answer

Now the crawler can request the page. The logs show a 200. Everyone relaxes.

They should open the HTML first. [Major AI crawlers do not reliably execute JavaScript](https://www.lifewood.ai/ai-crawlers.html). A browser can display pricing tiers, a feature grid, and a comparison table that a crawler never sees because those details arrive only after scripts run. Search Console showing a rendered page does not prove the AI fetcher received the same content.

This is the part that feels uncomfortably familiar to me as a PPC operator. A page can look right to the person reviewing it and still fail the system making the decision. The content team sees a polished product page. The fetcher sees a shell. Both descriptions can be true.

In the composite, the company’s public pricing is no help if the prices are absent from the HTML the fetcher receives. **A successful request is not proof of a readable answer.** For pages the business needs quoted, I would put the important facts in server-rendered or static HTML and verify them in the source, not just in a browser window. That check comes before a meeting about whether the pricing-page headline sounds more authoritative.

![A blank pricing table seen by an AI crawler versus the full table seen in a browser](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/9ea223cb-4fba-402c-a0a8-c38fd4d3c869.png)

## Fourth break: the readable page makes the answer hard to find

Only now does the content itself become the likely problem. The fetcher gets through. The important facts appear in the HTML. But the page spends its opening paragraphs introducing a category instead of answering the buyer’s question.

That is where a rewrite earns its place. [A citation-friendly structure puts a direct answer under the H1, uses headings for questions about matters such as cost and integrations, and states facts as numbers, names, and dates](https://www.okara.ai/blog/how-to-get-cited-by-perplexity). I used to tell clients to build authority with long intros. I was wrong when those intros kept the useful answer out of reach.

This does not mean turning every page into a stack of robotic FAQs. It means saying the useful thing while the reader is still there. An honest older price is also better than putting a 2026 date above 2024 prices and calling the page fresh. The page has to be both readable and worth quoting.

Product pages deserve that attention. Across buyer prompts, [product pages drew 24.1% of citations and articles drew 17.4%](https://cmomag.com/product-pages-get-24-of-ai-citations-study-finds/). In this composite, another blog brief would have been the team’s first response; the product page might have been the better place to look. **Once access works, fix the answer the buyer actually needs.**

## The earlier change nobody rechecked

The investigation now points backward. Before anyone ran those ten prompts, a redesign shipped. A security plugin updated. A developer tightened a firewall rule. The team kept publishing because nobody connected a site change to AI access.

This is why I would look backward through the change log, not just forward through the editorial calendar. A CDN rule can stop a live fetch even where robots.txt does not. An origin 403 can appear in a log while leaving the fetcher empty-handed. A JavaScript rebuild can preserve the page for browsers and remove the facts from the HTML. Each change looks small from the desk where it was made. Together they can explain why a brand disappears.

The defaults have changed over time, too; this [timeline from GPTBot to CDN defaults](https://groas.com/post/from-gptbot-to-default-blocking-a-dated) lays out that sequence. But the decision in front of this team is simpler than memorizing every bot name: **recheck access after the change ships**. Google still rendering a page is not that check.

## The fix: work from the edge to the answer

At this point, I would stop the next content brief long enough to establish where the failure sits. Not forever. Just until somebody can show what an AI fetcher receives. The order is short enough to use in a site-change review:

1. **Read the last 30 days of edge logs.** Look for OAI-SearchBot, ChatGPT-User, Claude-User, and PerplexityBot. Identify blocks and 403s before interpreting anything on the page.
2. **Inspect `/robots.txt` and the relevant crawler groups.** Check what the site permits for search discovery, what it blocks for training, and whether a managed rule appeared without the team’s knowledge.
3. **Read a money page with JavaScript off, then inspect its HTML.** If the price, comparison table, or product facts vanish, fix what the fetcher receives before polishing the prose around them.
4. **Read the first paragraph as a buyer would.** If it does not answer a real question clearly enough to quote, rewrite that opening and the headings beneath it.
5. **Match any break to what shipped.** After a redesign, CDN change, or plugin update, repeat the access check rather than assuming last month’s result still holds.

The full ordered pass is in this [21-point technical checklist](https://groas.com/post/the-ai-visibility-technical-checklist-21). For this postmortem, the important point is not the length of the checklist. It is that **the team checks the door before ordering more furniture**.

![A site-change logbook where one unannounced update erased AI access](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/ed2cff6f-e618-45f6-9d8d-986df710d357.png)

The PPC parallel still stings. When a landing page changed without warning, I could spend my time adjusting ads while the page the system needed had already changed. In this composite, the team could spend a year improving content while access or rendering kept buyers’ AI tools from using it. The work might be good. That is what makes the waste hard to spot.

My rule now is simple: **no new article gets briefed until access is proven**. Show that the relevant fetchers can get through, that the money page’s answer survives without JavaScript, and that its opening says something a buyer can use. Then write the next post.

## Frequently Asked Questions

### If my brand never shows up in ChatGPT or Perplexity answers, should I just publish more content?

Not before checking whether AI crawlers can actually reach your pages. New posts join old ones behind an access failure, and you cannot tell whether a post failed on its merits or was never available to the system answering the question.

### Do new Cloudflare domains block AI crawlers by default?

Yes. From 1 July 2025, Cloudflare domains block GPTBot, ClaudeBot and PerplexityBot by default. The site still works for human visitors and ranks in Google, so a content team may have no obvious reason to suspect the door closed.

### Does checking robots.txt catch every AI crawler block?

No. In one test of 314 live sites, 43% refused at least one AI crawler, but only 23 had a written robots.txt rule, with the other 112 refusing at the server. That is why edge logs should be checked for blocks and 403s, not just the site's robots file.

### If I block GPTBot in robots.txt, does that also stop ChatGPT from citing my site?

No. GPTBot collects training data, while OAI-SearchBot builds the ChatGPT search index, so they are different jobs needing separate, deliberate rules. Indexing, training and live access should be treated as separate concerns before calling a visibility problem solved.

### Do AI crawlers see content that only loads with JavaScript?

Usually not. Major AI crawlers do not reliably execute JavaScript, so pricing tiers, feature grids or comparison tables that arrive only after scripts run can be invisible to them. Put important facts in server-rendered or static HTML and verify them in the source, not just in a browser.

### Do AI answers cite product pages or blog articles more often?

Product pages. Across buyer prompts, product pages drew 24.1% of citations while articles drew 17.4%, so before briefing another blog post it may be worth checking whether the product page gives the answer a buyer's AI tool can quote.

### What should I check after a site redesign or CDN change to make sure AI visibility is unaffected?

Repeat the access check rather than assuming last month's result still holds. Look at recent edge logs for AI crawler blocks and 403s, recheck the robots.txt groups, and confirm the money page's key facts still appear in the HTML with JavaScript off. Google still rendering a page is not that check.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#webpage","url":"https://groas.com/post/postmortem-how-a-business-goes-invisible","name":"Postmortem: The Access Failure Behind a Brand’s Absence From AI Answers","description":"When a business disappears from ChatGPT and Perplexity answers, more content may not help. Trace the failure from crawler access to the page AI can actually read.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a351b684-3f64-40e5-96cd-a7e616ade69f.png"},"breadcrumb":{"@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#breadcrumb"},"dateModified":"2026-10-20T05:36:47.762Z","mainEntity":{"@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Postmortem: The Access Failure Behind a Brand’s Absence From AI Answers","item":"https://groas.com/post/postmortem-how-a-business-goes-invisible"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#article","url":"https://groas.com/post/postmortem-how-a-business-goes-invisible","mainEntityOfPage":{"@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Postmortem: The Access Failure Behind a Brand’s Absence From AI Answers","description":"When a business disappears from ChatGPT and Perplexity answers, more content may not help. Trace the failure from crawler access to the page AI can actually read.","datePublished":"2026-10-20T05:36:47.693Z","dateModified":"2026-10-20T05:36:47.762Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a351b684-3f64-40e5-96cd-a7e616ade69f.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"page, access, team, content, failure, answers, crawler, robots","wordCount":1855,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/postmortem-how-a-business-goes-invisible#faq","mainEntity":[{"@type":"Question","name":"If my brand never shows up in ChatGPT or Perplexity answers, should I just publish more content?","acceptedAnswer":{"@type":"Answer","text":"Not before checking whether AI crawlers can actually reach your pages. New posts join old ones behind an access failure, and you cannot tell whether a post failed on its merits or was never available to the system answering the question."}},{"@type":"Question","name":"Do new Cloudflare domains block AI crawlers by default?","acceptedAnswer":{"@type":"Answer","text":"Yes. From 1 July 2025, Cloudflare domains block GPTBot, ClaudeBot and PerplexityBot by default. The site still works for human visitors and ranks in Google, so a content team may have no obvious reason to suspect the door closed."}},{"@type":"Question","name":"Does checking robots.txt catch every AI crawler block?","acceptedAnswer":{"@type":"Answer","text":"No. In one test of 314 live sites, 43% refused at least one AI crawler, but only 23 had a written robots.txt rule, with the other 112 refusing at the server. That is why edge logs should be checked for blocks and 403s, not just the site's robots file."}},{"@type":"Question","name":"If I block GPTBot in robots.txt, does that also stop ChatGPT from citing my site?","acceptedAnswer":{"@type":"Answer","text":"No. GPTBot collects training data, while OAI-SearchBot builds the ChatGPT search index, so they are different jobs needing separate, deliberate rules. Indexing, training and live access should be treated as separate concerns before calling a visibility problem solved."}},{"@type":"Question","name":"Do AI crawlers see content that only loads with JavaScript?","acceptedAnswer":{"@type":"Answer","text":"Usually not. Major AI crawlers do not reliably execute JavaScript, so pricing tiers, feature grids or comparison tables that arrive only after scripts run can be invisible to them. Put important facts in server-rendered or static HTML and verify them in the source, not just in a browser."}},{"@type":"Question","name":"Do AI answers cite product pages or blog articles more often?","acceptedAnswer":{"@type":"Answer","text":"Product pages. Across buyer prompts, product pages drew 24.1% of citations while articles drew 17.4%, so before briefing another blog post it may be worth checking whether the product page gives the answer a buyer's AI tool can quote."}},{"@type":"Question","name":"What should I check after a site redesign or CDN change to make sure AI visibility is unaffected?","acceptedAnswer":{"@type":"Answer","text":"Repeat the access check rather than assuming last month's result still holds. Look at recent edge logs for AI crawler blocks and 403s, recheck the robots.txt groups, and confirm the money page's key facts still appear in the HTML with JavaScript off. Google still rendering a page is not that check."}}]}]}
```