“Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital.” - Aaron Levenstein
I’ve been very struck in the last week by a post by Tom Ewing, in which he said that the research debate should be about quality versus quantity rather than qualitative versus quantitative. I couldn’t agree more! Alastair Gordon has written in similar vein about the ‘myth’ of market research’s failure and that in most cases the failure is one of proper planning and thinking, rather than of research itself. Like any product or service, there are good and bad examples to be found. While good experiences are often (not always) associated with a price premium, they are more importantly associated with companies with the relevant expertise and the right approach to their customers.
These comments will seem (and are) trivial, but they need to be at the top of any research buyer’s mind. Quantity of research should never replace quality of thinking, which as Tom points out is all about framing questions in the right way, thinking through how data will be used, and understanding an issue from multiple perspectives. In the rest of Tom’s predictions for the coming year, there is a focus on the accessibility of larger and larger amounts of data and what this means for research One of his comments regards paradata, which Ray Poynter also references in his comments on the ‘exposome’, and there have been many other articles on the deluge or river or [insert metaphor] of data.
After reading the articles above, I referred back to a Boston Consulting Group report of 2009. One of their more interesting findings on the use of research, was that companies considered ‘best in class’ for their insight functions spent less per head on research than companies with less developed functions. Put another way, companies making good use of research (as measured by ROI), actually invest more resource in thinking and analysis and less resource in data collection itself.
With the ever-deepening data river, there is a tendency for researchers to revert to type and focus on quantity over quality. There’s always been an obsession with statistical significance in the industry (and in business generally) over the importance and relevance of findings. The reality is that as we have more and more data, researchers will find more and more ‘significant’ differences in their data. The question we all need to ask is how relevant and important are such findings? It’s great having your hand in the river, but you need to think first about what you want to pick up.
Data is only useful in as far as we apply critical thinking to answer relevant business questions, by formulating and testing clear hypotheses. In addition to any quantitative data we have, we need to frame our data within an understanding of consumer motivations as well as understanding of culture and society. Market research is only useful in so far as it reveals fundamental truths about consumers, and for this numbers are not enough. The concept of significance should be replaced with the concept of ‘effect size’ (ie business impact), and a focus on identifying data findings which can make a significant impact on our client’s business. And we should be clear that business impact only comes from changing behaviour (of customers or businesses), and that needs emotional connection as well as quantitative understanding.
As many have written before me, let’s not get carried away with the quantity of our data, and let’s focus instead on the quality of our thinking.