Most AI visibility scores are one screenshot dressed as a measurement. Ask the same buying question again and the brand list may change, so I would not make a decision from one run any more than I would judge ad copy on one day of data. The useful check is a fixed prompt set, repeated runs and one competitor logged as a control. This checklist covers ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews without requiring a paid visibility tool. Set it up once, run it on a schedule and keep the answers, not just a score. The test-protocol section tells you what to do when a page edit appears to move the needle.

Set up the panel once

  • Choose buyer-worded prompts. Pull 15 to 25 from sales calls, support tickets and RFPs. Include brand questions (“is [you] any good for X”), category comparisons (“best X for Y under $Z”) and problem questions (“how do I fix X without Y”). A panel of questions nobody asks gives you a tidy spreadsheet and very little else; the practical starting range is 10 to 30 prompts.
  • Freeze every prompt. Keep the words identical from week to week, including the qualifiers that make a question useful. Changing “best X for Y” to “best X for Y under $Z” may be sensible research, but it is a new question, not an improvement in your old trend. Add it separately rather than rewriting the panel to chase a better answer.
  • Pick one competitor as the control brand. Log that brand on the same prompts and engines as yours, including prompts where neither appears. A shared rise or fall can point to engine-wide movement; it does not, by itself, prove that either company changed its visibility. The comparison is there to challenge your first explanation, not to give you a second score to admire.
  • Build the log before you run anything. Use columns for prompt, engine, date, run, your brand named Y/N, control brand named Y/N, cited URL, position, tone and accuracy notes. Keep named and cited separate: an answer can name a brand without linking to it or cite a page without naming the site. The example log structure is a useful starting point, but the exact answer and source URL matter more than a polished dashboard.

Run the weekly check: steps 1 to 7

  1. Use the same surfaces each week. Run the panel on ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews; record each surface by name. Do not replace one halfway through the month and present the combined results as a trend. Different engines can move differently, which is one reason a single blended visibility number tells me less than it promises.
  2. Hold the run conditions steady. Use a fresh chat for each run, stay logged out where possible, and keep the same location, weekday and approximate time. Do not ask a follow-up in the same thread and call it a repeat. Location, history and personalization can change what comes back, so the aim is to avoid changing the setup at the same time as the answer.
  3. Repeat important prompts and count appearances. Run each important prompt three times per engine and record 2 of 3 rather than a single Y/N. If you choose a different repeat count, keep it fixed and show the denominator. The same prompt can produce different answers minutes apart; repetition makes that variation visible instead of letting one lucky answer masquerade as a score.
  4. Separate named from cited. Mark whether the answer names your brand and whether it links to one of your pages. These are different observations, not two spellings of “visible.” A brand can appear without a link, while a page can supply a fact without its publisher being named. Count both so a gain in mentions does not hide a loss in citations, or vice versa.
  5. Record the cited URL, not just the domain. Note whether it belongs to you, the control brand, a directory, a review site or another third party. “We got cited” leaves out the question that leads to a fix: which page did the engine use? If it is an old page or someone else’s description of your offer, keep that detail attached to the run.
  6. Note position and tone alongside the raw answer. A brand in the opening recommendation is not presented like one buried later in a list, and a polite warning is not an endorsement. Keep the answer text where you can reread it; a checkbox cannot preserve what the engine actually said about your price, fit or features.
  7. Report each engine separately. Calculate ChatGPT against ChatGPT and Gemini against Gemini before looking for any wider pattern. Never average their answers into a single result and then try to diagnose a page from it. If one engine names you and another cites the control brand’s help page, the useful information is in that difference.

Turn the log into decisions: steps 8 to 14

  1. Compare appearance rates with the control brand on identical prompts and runs. If you appear in 4 of 15 answers and the control appears in 11 of 15, you have a gap worth investigating, not proof of why it exists. If both rates move in the same direction, check whether the movement persists before crediting your latest edit or blaming an engine change.
  2. Flag prompts where nobody gets cited. They may expose room for a useful page, but an answer without links is not evidence that the existing pages are thin or that a new one will win. Read the answer and check what it does provide before you put a page on the worklist. The flag is a place to investigate, not a ready-made content brief.
  3. Flag prompts where a third party is cited instead of you. Record whether that source is Reddit, a review site, a directory, Wikipedia or another page, then read what it says. If the answer depends on a third-party description, rewriting your own page alone may miss the problem. The first job is to identify where the description buyers see comes from.
  4. Map each meaningful miss to a page to write, a page to fix or a third-party listing to check. Few mentions can point toward missing coverage or third-party trust; mentions without citations can point toward a retrieval or page-structure problem. Treat those as working diagnoses, then inspect the actual answers and URLs before assigning work. This mention-versus-citation split is useful because the two misses need not have the same fix.
  5. Log accuracy whenever your brand is named. Copy the precise description and mark a stale price, wrong feature or discontinued plan rather than writing “inaccurate.” This is easy to skip when the mention count looks good. A confident error may be more damaging than no mention, because the reader can act on the wrong description. Keep the wording and its source, if one appears, so someone can correct the right page or listing.
  6. Score appearances without hiding the denominator. Brand visibility rate = answers naming your brand ÷ total answers × 100; 22 of 100 is 22%. Calculate citation rate separately, and compare your appearance rate with the control brand’s on the same prompt set. Break the results out by engine. A neat percentage is only useful when you can still see which answers produced it.
  7. Check Google Search Console Performance for Google-side context, but do not use it as a substitute for this AI answer log. Keep the prompts, appearances and cited URLs in your own sheet. Search performance and what a particular generated answer says are not the same measurement; combining them into one score would put the uncertainty back in after you spent the week recording it.

Test a page edit without taking credit for churn

  • Question: Did the page fix change how often your brand appears or gets cited on the affected prompts, or did the answers move anyway? Write down the prompts and page before making the edit. If you choose them after seeing a favourable answer, you are no longer testing the question you started with.
  • Setup: Keep those prompts, engines, run conditions and repeat counts fixed. Run them on the same weekly schedule before and after the edit, and log named appearances and citations separately. Do not switch to “days cited out of seven” unless you also switch to a daily collection schedule; three runs on one weekday cannot answer a seven-day question.
  • Controls: Keep recording the competitor on the identical prompts and retain prompts where nobody gets cited. The control is imperfect: a competitor can change its pages too. Its job is to show whether your apparent gain is unusually specific, not to certify the cause. Use the control-group approach before putting a win in a report.
  • Measure: Compare appearance and citation rates by engine across the repeated runs and subsequent weeks. An Ahrefs study of 37,500 citations over 20 days found that about 50% of AI Overview citations disappeared the next day, with more churn on some other surfaces; many returned within days. That mechanism is why I expect plenty of week-to-week jumps to reverse rather than turn into a stable gain.
  • Decision: If your rate rises while the control stays relatively flat across several weekly checks, investigate the page edit as a possible cause and keep the control running. If both move together or the gain vanishes, keep logging rather than claiming a win. Either result changes the next action: pursue a sustained, specific improvement, or leave the victory slide alone and gather more answers.

Lab notebook tracking AI brand mentions across repeated runs

Automate collection, not judgment

  • Automate fetching only after the manual log shows you which columns you need. A sheet-fed n8n-style loop can read fixed prompts, query the engines and write answers back; the draft’s estimate for 25 prompts a week is low single-digit dollars a month, plus model costs such as roughly $5 to $14 per 1,000 Perplexity Sonar requests. That is an optional convenience, not a requirement for the check. Keep the schedule, wording and stored answers intact when you automate.
  • Read descriptions yourself. A script can record that you were named, but the question is whether the description helps or hurts. “Cheap” may be an advantage in one answer and a poor fit in another. I would rather read the wording and decide what to fix than accept a clean chart that quietly counts a wrong price as a success.
  • Keep the engine-level answers visible after automation. If Gemini cites a competitor’s help page while ChatGPT names you without a link, an averaged score erases the actionable part. For the fuller manual routine, start with measuring AI search visibility without trusting one screenshot, then automate the copying, not the interpretation.

Ink cartoon of a marketer buried in spreadsheets while a small robot fetches chatbot answers

Know when the sheet is the bottleneck

  • Stay manual while the panel and the number of people using it are manageable. The value of the sheet is that you can inspect the answer behind every count. Do not buy a dashboard because a screenshot looked persuasive; the point of this routine is to find out whether the pattern survives another run.
  • Move collection to a sheet-fed loop or paid report when more prompts, cities or brands make the routine hard to sustain. Keep the fixed panel and the answer-level log through the move. Otherwise you have traded a messy spreadsheet for a polished version of the same single-run problem.
  • Pay for execution when the misses are known but nobody is fixing them. A tracker can show that Reddit appears across eight prompts; it cannot, by itself, repair your page, address a third-party description or correct stale pricing. That is where groas is the better fit than another score dashboard: its SEO and AI Search service ships pages and technical fixes rather than leaving your team with a list of work.
  • Do not skip step 12: read what the answer says about you. A reported 10 to 20% of brand-specific answers in enterprise audits contain factual errors. Check each description, correct the source page or listing you can identify, and keep watching the repeated answers before claiming a fix worked. Skip this item and your visibility rate can rise while buyers receive the wrong price or a feature you no longer offer. In PPC terms, that is paying for more clicks that never become pipeline.