The number moved. That is not a result.
Two readings differ reportably only when their confidence intervals do not overlap. Everything else is the same reading, taken twice, with a story attached.
Somebody changes something, the figure moves four points, and a quarter of work gets declared successful. Sometimes it was. Usually nobody can tell, because nobody measured how much the figure moves on its own.
The test
- Take at least three runs per reading. Below three there is no interval and therefore nothing to compare.
- Compute the interval, not just the mean.
- Compare intervals. If they overlap, you have not measured a difference, whatever the two centre points look like.
- Report the coverage alongside, so a reading taken on eighty percent of the calls is not silently compared with one taken on all of them.
That is the whole method. It is not sophisticated; it is just unpopular, because it deletes most of the movement people want to talk about.
How wide is normal
Wider than intuition suggests. In a category we measured, a brand at 17.6% weighted share carried an interval from 15.1% to 20.4% across ten samples, with nothing in the world changing between them. Roughly five points of width, on a surface where a five point improvement is a headline.
What to do when the intervals overlap
Say so, and look at the things that move more slowly. The set of brands named at all changes less often than the percentage and means more when it does — a new name is a competitor arriving. The set of cited sources is the same: a new source is where the next answer is being decided.
Both are countable, both are checkable against the stored answers, and neither needs a statistical argument to be believed.
The short answers
How do I know if a change in visibility is significant?
Compare confidence intervals rather than point figures. Two readings differ reportably only when their intervals do not overlap, which requires at least three runs per reading.
How much does the number move on its own?
In a category we measured, a brand at 17.6% weighted share carried an interval from 15.1% to 20.4% across ten samples with nothing changing. About five points of width, on a surface where five points is treated as a headline.
Why three runs minimum?
Because below three there is no interval to compute, so there is nothing to compare and no way to distinguish a change from ordinary variation.
What if the intervals overlap?
Report that they overlap and look at slower-moving things: the set of brands being named at all, and the set of sources being cited. Both change less often and mean more when they do.
Does coverage need to be reported too?
Yes. A reading taken on eighty percent of its calls should not be silently compared with one taken on all of them, which is why coverage sits next to the figure rather than in a footnote.
How often should I report?
Monthly for most categories. If the interval is five points wide and the cadence is weekly, most of the meetings are about noise.
Find out in under ten minutes
One free check, no account and no card. You get the answers the assistants gave, not a score.
Every figure quoted here comes from a real question put to a real assistant, saved with the date and the assistant that answered it. This piece was written to answer the search "is my seo change significant".