---
title: "How to Track AI Search Visibility In-House Without Chasing a Score"
description: "Build a useful AI search report with fixed buyer prompts, repeated runs, and a weekly diff of brands named and pages cited. Automate collection; keep the judgment human."
url: "https://groas.com/post/how-to-automate-ai-search-visibility-rep-2"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/35529a2b-c1d6-4fea-a359-edc24f1b7ee0.png"
published: "2026-10-25T05:32:32.581Z"
modified: "2026-10-25T05:32:32.648Z"
---

October 25, 2026 · 11 min read

# How to Track AI Search Visibility In-House Without Chasing a Score

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![A tiny crowd stares anxiously at a giant wobbling gauge, while one small figure ignores it and shines a flashlight on a red-circled spot on a printed page.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/35529a2b-c1d6-4fea-a359-edc24f1b7ee0.png)

In this article

1. [Why does one check per prompt tell me so little?](#why-does-one-check-per-prompt-tell-me-so-little)
2. [Step 1: Which prompts should I track?](#step-1-which-prompts-should-i-track)
3. [Step 2: Which engines should I check, and how often?](#step-2-which-engines-should-i-check-and-how-often)
4. [Step 3: What should each row record?](#step-3-what-should-each-row-record)
5. [Step 4: What can I automate without losing the evidence?](#step-4-what-can-i-automate-without-losing-the-evidence)
6. [Step 5: What should I look for every Monday?](#step-5-what-should-i-look-for-every-monday)
7. [Step 6: Which part should I refuse to automate?](#step-6-which-part-should-i-refuse-to-automate)
8. [When should I stop doing this in-house?](#when-should-i-stop-doing-this-in-house)

Your AI visibility report should tell you which page to fix, not give you a number to worry about. Most in-house setups do the opposite: they test each prompt once, blend the results into a score, and watch that score wobble every week. I ran Google Ads accounts long enough to recognize the mistake. It is rank tracking copied onto a system that does not rank.

I would build the report like a Google Ads search terms report instead. Keep a fixed list of buyer prompts grouped like ad groups. Run each prompt several times so one odd answer does not send you after the wrong problem. Then read the weekly diff for two facts: **who got named and which page got cited**. Automate collection. Keep a human on interpretation. Skip that division of labor and you get a chart, not a decision.

## Why does one check per prompt tell me so little?

I used to tell clients a rank was a fact. Type the keyword, write down where you sit, move on. That habit breaks on ChatGPT or Gemini. In one large test with 600 volunteers running 12 prompts almost 3,000 times, [fewer than 1 in 100 pairs produced the same brand list](https://www.rivalhound.com/blog/ai-recommendation-variability/), and fewer than 1 in 1,000 matched in the same order. Same prompt, same engine, different answer ten minutes later. **One run is not a measurement.** It is a snapshot with unusually convincing formatting.

The second mistake is what you count. Teams blend everything into one visibility score, then wonder why the score moves while nothing changes in the business. Keep three questions apart: [Are you mentioned or cited in answers? Do people click through from AI? Can AI crawlers fetch your pages?](https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results) A rising mention rate can sit next to flat AI referral traffic for months. A single score hides that split. A search terms report would not replace its queries, clicks, and conversions with one mysterious dial.

That distinction sets the job for this report. It is not supposed to prove that AI search generated revenue from a handful of answers. It is supposed to show where your brand appears, where it disappears, and what the answers point to. **Use the report to choose an investigation, not declare a victory.**

## Step 1: Which prompts should I track?

Start with 20 to 30 real buyer prompts, not brand-polished queries. [Cover direct brand questions, category and comparison questions, and problem questions asked before buyers know you](https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results). That last bucket is where gaps hide. It catches the buyer who types “how do I stop paying for emergency callouts” instead of your service name. Pull wording from sales calls, support tickets, Search Console queries, and the communities where buyers complain. Write each prompt exactly as a buyer would type it.

Group the prompts the way you would group ad groups: discovery, comparison, problem, brand, and local if location matters. [Score brand prompts separately](https://medium.com/@citedgeo/how-we-measure-ai-search-visibility-citedgeo-5864b556cd9d). You will almost always win your own name, and that win can mask losses everywhere else. If five brand questions look healthy while five problem questions never name you, an average across all ten is not good news. It is camouflage.

Freeze the wording once you start. Keep a small exploratory set you rotate monthly if you want to test new phrasing, but leave the core set untouched so week-to-week comparisons mean something. Label the two sets clearly in your sheet. Otherwise, a newly worded question can look like a visibility change when you have really changed the test. **If the prompt text changes, the trend breaks.** Skip the temptation to add every interesting question to the core panel on Friday.

## Step 2: Which engines should I check, and how often?

For most small businesses, I would run ChatGPT, Gemini, and Perplexity weekly, then manually check Google AI Overviews for the ten highest-value prompts. That split has a practical reason: [Google AI Overviews has no official API and stays out of API-based collection](https://github.com/plutunieer/ai-visibility). Automate the first three and sample the fourth by hand in a logged-out browser. Record it as a separate engine, not as another run from the same system.

[API answers can also differ from what buyers see in the apps because of system prompts, personalization, location, and model versions](https://github.com/plutunieer/ai-visibility). That does not make collection useless. It makes the conditions of collection worth recording and the numbers worth treating as trends, not truth about every buyer’s screen. Do not put an API answer and a logged-out browser answer in the same pile, then call the pile an average.

**Run each prompt three times per engine per cycle.** That three-run floor follows a practitioner setup that logs [20 to 30 prompts times 3 runs, one row per answer](https://medium.com/@citedgeo/how-we-measure-ai-search-visibility-citedgeo-5864b556cd9d). It is a cheap way to stop chasing ghosts. A prompt you win three out of three times looks steadier than one you win once out of three. Neither deserves to be described as a permanent rank.

Report mention rate, citation rate, and share of voice by engine, not as one blended score. A weekly cycle is enough for almost everyone reading this. More frequent collection is easy to schedule; more frequent interpretation is just another meeting unless someone will act on what changed.

## Step 3: What should each row record?

**Keep one row per answer, not one row per prompt.** With 25 prompts, three engines, and three runs each, that gives you 225 rows a week. The volume is the point: you can return to the answer behind a change instead of trusting a cell someone marked yes.

At minimum, log:

- Date, engine, prompt ID, exact prompt text, and run number.
- Brands named in order and domains cited, including the specific cited URL.
- Separate yes-or-no fields for whether your brand was mentioned, cited, and recommended.
- Competitors cited, citation position, and a notes field for refusals, stale answers, or anything else the columns cannot explain.

A [spreadsheet with Query, Platform, Date, Cited Y/N, Citation position, Competitors cited, and Link provided](https://frase.io/blog/ai-search-tracking-monitor-your-visibility-across-ai-engines) covers much of the minimum. Keep the brand-name and cited-page fields distinct. A domain appearing as a source is not the same thing as your brand appearing in the prose, and neither tells you which of your pages earned the link unless you save the URL.

That distinction matters in the answers themselves. [Semrush data cited by Adriel found 62% of AI citations were _ghost citations_: a domain was sourced but the brand was never named](https://www.adriel.com/blog/how-to-get-your-brand-cited-in-ai-search-a-practical-guide-for-enterprise-marketers). The reported engine split was stark, too: ChatGPT cited a source 87% of the time but named the brand in about 20.7% of answers, while Gemini named brands 83.7% of the time but linked only 21.4%. **If you track only mentions, you miss the citation gap.** If you track only cited domains, you miss what the answer actually says about you.

## Step 4: What can I automate without losing the evidence?

I run collection with a sheet, a scheduler, and three APIs. [The sheet holds the prompt list; a job reads each row, sends it to ChatGPT, Gemini, and Perplexity, checks each answer for the brand name, and writes the result back on schedule](https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results). Check the brand match before trusting that yes-or-no field. “Apple Plumbing” should not win because an answer discussed apple the fruit.

The request cost can stay small for a lean list. Perplexity Sonar search runs roughly $5 to $14 per 1,000 requests. At 25 prompts, three runs, and four weekly cycles, that is about 300 Perplexity requests a month, before accounting for the other providers or the time spent building and reading the report. Cheap collection is not free reporting.

One [open-source version of the setup](https://github.com/plutunieer/ai-visibility) shows the shape: a config file for the brand, aliases, domains, competitors, and fixed prompts, plus a script that sends prompts to providers with web search on and logs whether the brand was mentioned, its rank among tracked brands, its cited domain, and competitors. Its eight-prompt, three-provider setup runs daily as a scheduled job for a few cents per day. You do not need that exact script. You need a repeatable collection job and an answer you can inspect afterward.

API collection misses AI Overviews, and browser checks do not scale, so use both for what each does well. Automate the three APIs. Sample Overviews by hand for your ten money prompts in a logged-out session, and paste those answers into the same sheet with their engine clearly marked. Most important: **store the full answer text, not just the yes or no.** Re-reading is how you notice that a competitor’s cited page has a pricing table yours lacks. If you want the Sheet and script shape, this [weekly AI brand monitor with Sheets, two APIs, and 25 buyer prompts](https://groas.com/post/build-your-own-ai-brand-monitor-in-an-af) maps to the workflow.

![Handmade model of a weekly AI monitoring assembly line sorting prompt cards into trays](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/56c408fd-ee12-496d-8e21-9654510cd7a5.png)

## Step 5: What should I look for every Monday?

I sort the fixed prompts into three buckets before I look at any aggregate rate:

- **Gained:** We appear in at least two of three runs after missing last week.
- **Lost:** We disappear after previously holding.
- **Never-had:** We have never appeared, and the same competitor keeps winning.

The third bucket matters more than it looks. A competitor that owns the same five problem prompts for six straight weeks is pointing you toward the pages to examine. Raw counts move; the bucket list tells you where to read. Keep the engine and prompt group beside each change so a brand-query gain cannot distract you from a problem-query loss.

Noise looks like a prompt flipping from win to loss on a single run, then flipping back next cycle, with no change in cited URLs. I ignore it. Movement worth investigating looks like a shift across two consecutive weeks with a new cited domain attached, or a mention rate dropping from 80% to 20% on the same fixed wording. That is why the fixed panel and repeat runs exist. This breakdown of [how to measure AI search visibility without trusting one screenshot](https://groas.com/post/how-to-measure-your-brand-s-visibility-i-2) applies the same rule to screenshots.

**Two weeks plus a URL change means work. One lonely flip means weather.** The work starts with reading the answers, not filing a ticket to rewrite the site.

![Weekly diff ledger showing gained, lost, and never-held AI prompts](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/396fe2b4-ee3e-4d83-bc6d-27bc76339334.png)

## Step 6: Which part should I refuse to automate?

Collection runs on a schedule. Interpretation does not. When a prompt moves from gained to lost, read the raw answers before touching a page. I look for which competitor replaced me, which URL the answer cited, and what that page contains that mine lacks: a price, a service-area list, a comparison table, or a direct answer in the first 100 words. The cited page gives you something concrete to inspect. It does not, by itself, prove why the engine chose it.

Then make one page change tied to the gap and log the date beside the prompt. Next week’s diff can support or challenge that decision. If you change the page, the prompt wording, and the way you classify mentions at the same time, you will not know what the next report is telling you. **Automate the collection; keep a human responsible for the fix.** Automating that judgment is how teams rewrite five pages because of one noisy answer.

![Cartoon of a marketer watching a conveyor belt of AI answers while a manual lever marked fix page waits untouched](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/fd23c49a-5e27-4bd9-840c-fbb76ff79725.png)

## When should I stop doing this in-house?

The script is cheap. The reading is not. A 25-prompt setup takes me four to six hours to build the first time: writing the prompts, wiring the sheet, and testing matches so “Apple Plumbing” does not match apple the fruit. After that, allow about 60 to 90 minutes a week to read the diff properly, plus another hour a month when a model update breaks a parser or an API key expires. **Count 6 to 8 hours a month before you call it free.** Say you are spending $20k a month on search and your time is worth $150 an hour. That is $900 to $1,200 in labor to keep the report honest.

Skip DIY entirely if you have more than 50 prompts, more than three locations with different prompt sets, or nobody who will actually open the sheet on Monday. I have watched all three fail the same way: the job runs, nobody reads it, and three months later someone asks why the score dropped in April. If collection runs but interpretation never happens, you paid for noise.

That is where this comparison of [DIY checks versus trackers versus done-for-you AEO](https://groas.com/post/diy-script-vs-visibility-tracker-vs-done) earns its keep. It shows what changes when the same prompt panel is watched every day, the page fix ships the same week, and every change lands in a log with its reason. groas runs that way on purpose: continuous execution with one human owning the outcome, not a dashboard waiting for you to log in.

**Start today with 15 prompts, not 30.** Pick five brand, five comparison, and five problem questions straight from your last month of sales calls. Run each one three times in ChatGPT and paste the full answers into a sheet with columns for mentioned, cited URL, and competitors named. That afternoon tells you more than a blended score because you can see which competitor keeps showing up and which of your pages never gets cited. Automate the runs next week. Keep the reading for yourself until the reading costs more than handing it over. That is when done-for-you stops being a luxury and starts being math.

## Frequently Asked Questions

### Do I need to run an AI search prompt more than once?

Yes. In a test of 600 volunteers running 12 prompts almost 3,000 times, fewer than 1 in 100 pairs produced the same brand list, and fewer than 1 in 1,000 matched in the same order. One run is a snapshot, not a measurement, so run each prompt several times before acting.

### Why shouldn't I use one blended AI visibility score?

A single score hides the difference between three separate questions: whether you are mentioned or cited in answers, whether people click through from AI, and whether AI crawlers can fetch your pages. A rising mention rate can sit next to flat AI referral traffic for months, and an average will not show that split.

### How many AI search prompts should I track and where do I get them?

Start with 20 to 30 real buyer prompts covering brand questions, category and comparison questions, and problem questions asked before buyers know you. Pull wording from sales calls, support tickets, Search Console queries, and buyer communities. Group them like ad groups, score brand prompts separately, and freeze the wording so week-to-week comparisons mean something.

### Which AI engines should I check and how often?

For most small businesses, run ChatGPT, Gemini, and Perplexity weekly, and manually check Google AI Overviews for your ten highest-value prompts in a logged-out browser. Run each prompt three times per engine per cycle, and report mention rate, citation rate, and share of voice by engine rather than as one blended score.

### What should I record when tracking AI search visibility?

Keep one row per answer, logging the date, engine, prompt ID, exact prompt text, run number, brands named in order, cited domains and URLs, competitors, citation position, and separate fields for mentioned, cited, and recommended. Keep brand mentions and cited pages distinct: Semrush data cited by Adriel found 62% of AI citations were ghost citations, where a domain was sourced but the brand was never named.

### Which parts of AI visibility tracking can I automate?

Automate collection: a sheet, a scheduler, and APIs can send prompts to ChatGPT, Gemini, and Perplexity, check answers for your brand name, and log results on schedule. Keep interpretation and fixes human. When a prompt moves, read the raw answers and the cited page before making one page change tied to the gap, and log the change date beside the prompt.

### How do I tell real changes from noise in my AI visibility report?

Ignore a single run that flips a prompt from win to loss and back with no change in cited URLs. Movement worth investigating is a shift across two consecutive weeks with a new cited domain attached, or a mention rate dropping from 80% to 20% on the same fixed wording. Start by reading the answers, not rewriting pages.

### How much time does in-house AI visibility tracking take?

A 25-prompt setup takes four to six hours to build the first time, then about 60 to 90 minutes a week to read the diff plus roughly an hour a month for maintenance, so count on 6 to 8 hours a month. Skip DIY if you have more than 50 prompts, more than three locations with different prompt sets, or nobody who will actually read the results.

## Paid Ads & SEO, Run For You

Businesses pay one flat monthly fee per service, agencies resell it under their own brand, and groas does the work either way.

[See Pricing](https://groas.com/pricing)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","description":"groas is a done-for-you marketing service, not software: elite people working with proprietary technology systems run search ads, Meta ads, SEO and AI search for businesses, and white label for agencies, with a dedicated senior strategist owning every account.","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#webpage","url":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2","name":"How to Track AI Search Visibility In-House Without Chasing a Score","description":"Build a useful AI search report with fixed buyer prompts, repeated runs, and a weekly diff of brands named and pages cited. Automate collection; keep the judgment human.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/35529a2b-c1d6-4fea-a359-edc24f1b7ee0.png"},"breadcrumb":{"@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#breadcrumb"},"dateModified":"2026-10-25T05:32:32.648Z","mainEntity":{"@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"How to Track AI Search Visibility In-House Without Chasing a Score","item":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#article","url":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2","mainEntityOfPage":{"@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"How to Track AI Search Visibility In-House Without Chasing a Score","description":"Build a useful AI search report with fixed buyer prompts, repeated runs, and a weekly diff of brands named and pages cited. Automate collection; keep the judgment human.","datePublished":"2026-10-25T05:32:32.581Z","dateModified":"2026-10-25T05:32:32.648Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/35529a2b-c1d6-4fea-a359-edc24f1b7ee0.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"prompts, cited, search, prompt, brand, score, answer, collection","wordCount":2222,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"My background includes Goldman Sachs and Stanford Computer Science.","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/how-to-automate-ai-search-visibility-rep-2#faq","mainEntity":[{"@type":"Question","name":"Do I need to run an AI search prompt more than once?","acceptedAnswer":{"@type":"Answer","text":"Yes. In a test of 600 volunteers running 12 prompts almost 3,000 times, fewer than 1 in 100 pairs produced the same brand list, and fewer than 1 in 1,000 matched in the same order. One run is a snapshot, not a measurement, so run each prompt several times before acting."}},{"@type":"Question","name":"Why shouldn't I use one blended AI visibility score?","acceptedAnswer":{"@type":"Answer","text":"A single score hides the difference between three separate questions: whether you are mentioned or cited in answers, whether people click through from AI, and whether AI crawlers can fetch your pages. A rising mention rate can sit next to flat AI referral traffic for months, and an average will not show that split."}},{"@type":"Question","name":"How many AI search prompts should I track and where do I get them?","acceptedAnswer":{"@type":"Answer","text":"Start with 20 to 30 real buyer prompts covering brand questions, category and comparison questions, and problem questions asked before buyers know you. Pull wording from sales calls, support tickets, Search Console queries, and buyer communities. Group them like ad groups, score brand prompts separately, and freeze the wording so week-to-week comparisons mean something."}},{"@type":"Question","name":"Which AI engines should I check and how often?","acceptedAnswer":{"@type":"Answer","text":"For most small businesses, run ChatGPT, Gemini, and Perplexity weekly, and manually check Google AI Overviews for your ten highest-value prompts in a logged-out browser. Run each prompt three times per engine per cycle, and report mention rate, citation rate, and share of voice by engine rather than as one blended score."}},{"@type":"Question","name":"What should I record when tracking AI search visibility?","acceptedAnswer":{"@type":"Answer","text":"Keep one row per answer, logging the date, engine, prompt ID, exact prompt text, run number, brands named in order, cited domains and URLs, competitors, citation position, and separate fields for mentioned, cited, and recommended. Keep brand mentions and cited pages distinct: Semrush data cited by Adriel found 62% of AI citations were ghost citations, where a domain was sourced but the brand was never named."}},{"@type":"Question","name":"Which parts of AI visibility tracking can I automate?","acceptedAnswer":{"@type":"Answer","text":"Automate collection: a sheet, a scheduler, and APIs can send prompts to ChatGPT, Gemini, and Perplexity, check answers for your brand name, and log results on schedule. Keep interpretation and fixes human. When a prompt moves, read the raw answers and the cited page before making one page change tied to the gap, and log the change date beside the prompt."}},{"@type":"Question","name":"How do I tell real changes from noise in my AI visibility report?","acceptedAnswer":{"@type":"Answer","text":"Ignore a single run that flips a prompt from win to loss and back with no change in cited URLs. Movement worth investigating is a shift across two consecutive weeks with a new cited domain attached, or a mention rate dropping from 80% to 20% on the same fixed wording. Start by reading the answers, not rewriting pages."}},{"@type":"Question","name":"How much time does in-house AI visibility tracking take?","acceptedAnswer":{"@type":"Answer","text":"A 25-prompt setup takes four to six hours to build the first time, then about 60 to 90 minutes a week to read the diff plus roughly an hour a month for maintenance, so count on 6 to 8 hours a month. Skip DIY if you have more than 50 prompts, more than three locations with different prompt sets, or nobody who will actually read the results."}}]}]}
```