“All models are wrong, some models are useful.” - George Box
The flaw of averages
Market research reports are generally packed full of average scores and comparisons of means, and not often enough with distributions and extremes. In the real (business) world this is a mistake as, in the words of Sam Savage’s (f)law of averages, ‘plans based on average assumptions are wrong on average’. To put this another way, errors occur in the real world when we replace uncertain numbers by single (or simple) averages.
Marketing is full of uncertain numbers, several of them outside the scope of most market research. In his book The Flaw of Averages, Sam Savage explores the flaws in looking only at average assumptions across a wide variety of cases. Although there are some similarities with The Black Swan, this book focuses much more on the concept of distributions and alternative scenarios and less so with the extreme events which Nassim Nicholas Taleb writes about. He also discusses the important difference between uncertainty and risk (quoting Donald Rumsfeld), the usefulness of decision trees, and the importance of distributing risk across portfolios among many other topics.
The reality is that the average tells us very little about almost everyone (or every case). If every family has 1.5 children on average, how do you design a baby buggy? By contrast, if half of families have 1 child and the other half 2 children, you know what you need to produce.
More generally, understanding the distribution of an item, event or response can help us understand the range of potential outcomes for anything you want to understand. In particular, by understanding the shape of the distribution (is it even, or skewed) and the range of responses (how wide is the range of possible outcomes), we can begin to model the range of possible (or probable) future scenarios. One of the common tools for modelling scenarios is monte carlo simulation which is discussed in the book and also on Wikipedia here.
Bells and bows
This is not the place to discuss basic statistics (there are many fine blogs which do), but the concept of a distribution of responses is the key. Around any ‘average’ there will be some higher and some lower responses, and the key to understanding how much the average tells you is to look at the shape and width of that distribution. If the shape is fat and wide, then the answer is that it tells you very little.
When we come to the real world of business and marketing, the problem gets compounded by the number of different variables interacting to produce a particular result or event. For example, take new product forecasting. Although researchers tend to focus on the consumer response to a new product, this is only one of the uncertainties which will effect the final success of a new product, which will of course also depend on awareness, distribution, the speed of diffusion, the market (competitive) context and so on. All of these variables have average (or predicted) outcomes as well as distributions which explain the range of likely outcomes. And the best way to model such a range of potential influences is to use monte carlo simulation. In such a multidimensional problem, the use of averages for each variable to produce an average predicted outcome is almost certain to be wrong (and potentially highly misleading).
In a complex world of interacting influences, it is virtually certain that at least one element or input will not go to plan. And where such complex interactions can be easily modelled on your laptop, it makes sense to take time to see the range of possible outcomes and where the weak links in the chain are likely to occur.
Averages are wrong. Simulation is useful.
The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty by Sam Savage (2009)
The Black Swan: The Impact of the Highly Improbable (2nd edition) by Nassim Nicholas Taleb (2010)