Answers move. Most of it means nothing.
The useful question is not whether AI answers change — they do, constantly — but how much of that movement is signal. Almost everyone reports the noise as progress.
Every tool in this category will tell you that AI answers change. Very few will tell you by how much, because that number is expensive to produce and unflattering to own: once you publish the size of ordinary variation, you can no longer report movement smaller than it as a result.
The size of ordinary drift
In a run against a local hospitality category, the leading brand was named in 91 of 165 answers, for a weighted share of 17.6%. Across ten samples the interval ran 15.1% to 20.4% — roughly five points wide, with nothing in the world having changed between samples.
That is the bar any claimed improvement has to clear. A brand reporting that it moved from 15% to 19% has reported the width of the noise.
What changes underneath
- The composition itself, which differs between identical asks and is the largest short-term source of movement.
- The pages the engine reads, which are edited by other people on their own schedule. A roundup updated last night moves your answer this morning.
- The model, which is replaced periodically and can shift a category’s answers wholesale on the day it ships.
- Your own presence in the sources, which is the only one you control and the slowest of the four.
The asymmetry there is the whole scheduling problem. The thing you can change moves in months. Three of the four things that move the number do so in days, and none of them is about you.
How often to measure
Monthly, for almost everybody. Weekly readings on a surface with a five point interval mostly produce meetings about noise, and daily readings produce a chart that looks like activity and contains none.
The exception is a deliberate change. If you have just landed on a roundup that gets cited constantly in your category, a reading before and a reading after is worth taking close together — but even then, three runs each side or the comparison is not one.
What to watch instead of the number
Two things move more slowly and mean more. The set of brands being named at all, because a new name entering your category’s answers is a competitor arriving, and the set of sources being cited, because a new source appearing is where the next answer is being decided. Both change less often than the percentage and both are actionable when they do.
The short answers
Do AI answers change over time?
Constantly, from four causes: the model composing differently on identical asks, the pages it reads being edited by other people, the model itself being replaced, and your own presence in those sources changing. Only the last is about you, and it is the slowest.
How much do AI answers vary without anything changing?
More than most people expect. In one of our runs a brand at 17.6% weighted share of voice carried a confidence interval from 15.1% to 20.4% across ten samples, with nothing in the world changing between them. Roughly five points of width is the bar any claimed improvement has to clear.
How often should I track AI visibility?
Monthly for almost everybody. Weekly readings on a surface with a five point interval mostly produce meetings about noise. Daily produces a chart that looks like activity and contains none.
How do I tell a real change from normal drift?
Compare intervals rather than points, which means at least three runs per reading. Two snapshots differ reportably only when their confidence intervals do not overlap.
What should I watch if the percentage is too noisy?
The set of brands named at all, and the set of sources cited. Both move more slowly than the share figure and both are actionable: a new name is a competitor arriving, and a new source is where the next answer is being decided.
Find out in under ten minutes
One free check, no account and no card. You get the answers the assistants gave, not a score.
Every figure quoted here comes from a real question put to a real assistant, saved with the date and the assistant that answered it. This piece was written to answer the search "do ai answers change".