A 14-point drop in an AI visibility report looks like a problem to fix. Before you change a page or buy a better-looking dashboard, find out whether your score can drop 14 points when you change nothing.

In one controlled set of 2,961 runs, ChatGPT and Google AI had a less than 1 in 100 chance of repeating the same brand list twice. I covered the premise in Your AI Visibility Score Isn’t a Rank. It’s a Sample. That is worth a second look because knowing a score is a sample does not tell you how much your number swings, or whether last week’s drop was real.

The question for this test: How much does your brand visibility move when you change nothing, and how large must a move be before you act? Run the repeats yourself first. The spread becomes your noise floor for judging both future results and the software that reports them.

Setup: freeze prompts tied to buying decisions

Start with 30 to 50 prompts you can trace to revenue. I would draw them from three places: phrases prospects actually use on sales calls, Search Console queries that already drive impressions, and paid search terms that have spent money. Then give the set a deliberate mix:

  • Category prompts: best inventory software for mid-market retail
  • Comparison prompts: X vs Y for warehouse teams
  • Problem prompts: how to cut stockouts across two warehouses

The mix matters because a wording change can swap the brand list. In one live demo, changing best to shortlist swapped one named tool for another in Perplexity (Outrank self-run test). Independent tool comparisons likewise recommend starting with 20 to 50 real-customer prompts. The prompt set deserves at least as much attention as the tool tracking it.

Write each prompt exactly as you will submit it. Give it an ID, put it in a sheet and freeze the wording for the test. Resist the urge to improve an awkward prompt halfway through. If it reflects a real buyer question, its awkwardness may be the point. If you decide it does not belong, remove it from the next test rather than silently changing this one. If the prompt list moves, the trend is fiction.

Controls: give each answer a clean start

Run each prompt cold: no follow-up in the same chat, the same location and language each time, and a fresh logged-out window where the product allows it. Note the model version when the engine shows one. Keep browsing settings consistent, and record the date and time even if you cannot control when retrieval changes.

These are not ceremonial lab rules. In-session memory, location, account state, live retrieval shifts and silent model updates can all move an answer without any change to your site (seven noise sources listed by Am I Visible). You cannot pin down every variable in ChatGPT, Gemini or AI Overviews. You can stop introducing avoidable ones yourself.

Use this checklist before the first run:

  1. Freeze the prompt text and IDs.
  2. Choose the engine, location, language and account state for each set.
  3. Decide whether browsing is on where that setting is available.
  4. Start every repeat in a fresh session rather than continuing the previous answer.
  5. Record any setting or model-version change you notice; do not quietly blend it into the baseline.

That gives you a defensible answer to a useful question: what varied while my setup stayed as steady as I could make it? It does not make the engines deterministic. Measuring that lack of determinism is the job.

Schedule five runs, not five different tests

Run each prompt at least five times per engine; use nine if you can spare the time. The accuracy guide I trust on this calls five the minimum and nine the better default. One run per prompt is one stochastic sample dressed up as a score.

For a five-run hand test, do two back-to-back runs in one sitting, then one run on each of the next three days. Keep the daily runs in a similar time window. The first pair gives you a glimpse of generation wobble; the later runs let retrieval and updates show themselves. If you choose nine repeats, keep the same controls and log when each repeat happened. Do not mix a nine-run result into a five-run comparison without saying so.

Small sets punish you fast. In Otterly’s 252,407-answer test, moving from 10 prompts to 100 cut brand coverage variation by about 73%. At 10 prompts, 90% of results landed anywhere from 39% to 71%. At 100 prompts, that band tightened to 51% to 60%. A short prompt list may be convenient, but convenience is not stability. Block time for 30 prompts × 5 runs × 3 engines: 450 answers.

For each answer, record whether the response names your brand and whether it cites a brand URL. Those are different events. In Otterly’s long stability run, only 12 to 19% of brands appeared on a single day out of 30, while up to 67% of cited sources did, and nearly half of 46,009 URLs were cited just once. Mentions persist differently from sources. A separate Semrush study of 3,981 brand appearances found 62% were citations where the answer never named the brand at all, with ChatGPT citing in 87% of appearances but naming the brand in only 20.7%.

Blend mention and citation into one visibility number and you lose the distinction you need later. A buyer may still hear your name while the engine rotates the link it shows. Or your URL may appear without the answer naming you. Those changes call for different investigations.

Keep one row per answer. Your sheet needs six fields:

  • Date and time of the run
  • Engine and model version, if shown
  • Prompt ID, tied to the frozen text
  • Brand mentioned: yes or no
  • Brand URL cited: yes or no
  • Competitors named and sources cited, when retrieval is live

Keep the raw answers too. A daily average alone hides whether one prompt produced five different brand lists or every prompt behaved consistently. The spread across repeats is the measurement, not a nuisance to smooth away.

Readout: set a threshold before the next report arrives

Calculate mention rate separately for each engine: brand mentions divided by answers for that engine. Then compare rates across your repeat windows. As an illustration, suppose ChatGPT names you in 62 of 150 answers. Your overall baseline is about 41%. If the daily slices range from roughly 36% to 47% while your site and prompts stay unchanged, the observed no-change swing is about 11 points. That is the floor this test found, not proof that every future 11-point move is harmless.

Set your action rule before you see the next score. I would ignore a week-to-week move smaller than the biggest no-change swing I measured, and investigate a move that clears that threshold two weeks running. That rule trades some speed for fewer false alarms. If a possible problem cannot wait two weeks, you can look sooner; just do not mistake an early look for confirmation. A 14-point drop means little until you know whether a 12-point drop can happen by itself.

Do not average the engines into a reassuring single figure. If a hypothetical ChatGPT series stays within a 5-point band while Gemini swings 18 points, give Gemini a wider threshold and more repeats. Do not prescribe a content rewrite because one volatile series had a bad day. If the same three prompts swing across every engine, inspect the prompts before blaming the models. best software for business can invite several intents; a question tied to one buyer and one job gives the next test a cleaner target. Tighten the wording, then start a new baseline rather than pretending the revised prompt is the old one.

My expectation is that at least some apparent gains and losses will fall inside the no-change range. The mechanism is straightforward: repeated generation and changing retrieval can alter which brands an answer includes, even when your marketing has not moved. The test tells you how much of that motion you see on your prompts. It cannot certify that a move outside the band was caused by your work. It tells you where investigation is worth the time.

Vendor check: make the software show its repeats

Take your baseline to a reporting-tool demo. Ask for the runs behind the score and the variance across them, not just the headline chart. Ask who wrote the prompts and whether you can replace them with yours. Then ask which conditions stay fixed: region, language, account state, model version, time window and cold-start versus continuing sessions. Finally, ask how the tool defines a mention and a citation.

Those questions follow the vendor vetting script from Am I Visible: raw prompt log, runs per prompt and variance, cold-start versus batched sessions, panel size and sourcing, account state, engine and model version, and mention definition. One run per prompt with no variance shown is a single stochastic sample with a logo on it. I would not buy it as evidence of a small week-to-week change.

Give the vendor 10 of your frozen prompts. Run those same prompts yourself five times each that week, then compare the ranges by engine. If its ChatGPT figure lands inside your no-change spread, the tool may be reporting motion your hand test already sees. If it lands far outside, ask what differs before deciding either result is wrong: prompts, region, browsing setting, account state or collection method. Scraped HTML and an official API, for instance, can return different answers according to Sona’s tool check. A disagreement is a reason to inspect the setup, not an invitation to choose the prettier number.

Automation earns its fee in two places: scale and consistency. Five runs of 30 prompts across three engines produce 450 answers. Doing that regularly by hand invites skipped repeats, reused chats and quiet prompt edits. Software does not get bored. It can, however, report the noise faster without measuring it better: a clean chart and daily cadence still leave you with single samples if there is only one run per prompt.

My buying rule is plain. If a tool keeps my prompts frozen, shows runs per prompt and plots a range beside the score, it can replace the spreadsheet. If it cannot, I still need the spreadsheet to judge what its chart means. For a closer look at choosing prompts, see the 30-prompt test for AI search visibility. Do not pay to automate a test you have not run once by hand.

Lab notebook showing AI brand-mention tallies varying across repeated runs

Follow-up: keep the floor current and investigate the exceptions

Once you have a floor, re-establish it once a quarter or after an announced retrieval or model change. Use the same prompts for a condensed week, with three runs per engine instead of five, to check whether the band has moved. Keep that shorter check labelled as a check; do not quietly swap its range for the fuller baseline if the results disagree. In that case, run more repeats.

Leadership does not need the 450-row sheet. Give them one line per engine: mention rate, usual no-change range and whether the new result cleared it. For example: ChatGPT 41%; usual range 36–47%; this month 44%; no action. Or: Gemini 28%; usual range 18–34%; this month 12%; investigate. Those are illustrative readouts, not reported results. Their value is that the decision sits next to the number.

When a move clears the floor two weeks running, work through the raw answers in order:

  1. Split the change by prompt. Did several category prompts fall together, or did one broad question pull down the average? A concentrated fall points you toward that intent; a spread fall calls for a wider look.
  2. Split mentions from citations. If answers still name you but your URLs disappear from the citations, investigate sourcing before calling it a brand collapse. Rewriting the homepage on the strength of a rotated citation is busywork.
  3. Check access. For AI Overviews, look for an indexed, Google-eligible page with a usable snippet. For ChatGPT Search, check whether OAI-SearchBot is allowed (Semrush crawler summary). A fetch block is an access problem. Treat it like one.

This is why a one-off audit has limited shelf life. Its baseline becomes harder to use when someone edits a prompt or an engine changes retrieval. The durable version keeps the same prompts running, repeats them, logs mentions and citations separately, plots the range next to the score and records why the setup changed. That is how groas runs it for clients: specialized models handle repeat sampling around the clock, while a named strategist sets the threshold and decides which moves warrant action. I still want the sheet. I just do not want someone pasting 450 answers into it every Friday.

Run the hand test before the next red-circled report. If a drop sits inside your measured band, hold off on the redesign, vendor switch or urgent fix and keep watching. If it clears the band two weeks running, split it by prompt, mention and citation, then investigate what changed. Either result gives you a better next decision than a score with no error bar. Measure the noise, then decide what the move means.