Answers without Questions

Sep 13 2012

I believe that market research still relies far too much on direct questions and answers and that many current practices in research suffer from serious flaws. There are three big problems in relying on direct question and answer approaches:

  1. They do not take account of the different systems in the brain
  2. They too often ignore the importance of context (and it’s role in memory)
  3. The act of asking questions can itself change the answers you get

Why do I say that questions and answers do not take account of the different systems in the brain? Daniel Kahneman describes the profound differences between what he calls System 1 and System 2. And many others have described the differences between unconscious and conscious thinking or implicit and explicit knowledge. These differences are real and profound. As Daniel Kahneman says, “We are to thinking as cats are to swimming, we can do it if we have to.”

It is estimated that System 1 (implicit) is processing around 11 million pieces of information every second – this is the information coming from the world around us via the senses. System 1 is our experiencing brain.

By contrast, System 2 (explicit) can process only 50 pieces of information per second by the most generous of estimates. System 2 is our remembering brain. It can be described as the “spin doctor” of our experience. System 2 likes to make a consistent worldview from the conflicting drives and contradictory behaviours that make us human.

The differences between the systems are summarized in the table below.

Multiple systems Single system
On-line pattern detection After the fact (check and balance)
Here and now Taking the long view
Automatic (fast, unintentional, uncontrollable, efficient, intuitive) Controlled (slow, intentional, cognitive, energy sapping)
Rigid Flexible
Implicit Explicit
Sensitive to negative information Sensitive to positive information
Hard working Lazy
Easy problems Hard problems

As you can see our unconscious mind is fast, unintentional, uncontrollable, efficient and intuitive, while our conscious mind is slow, intentional, cognitive and energy sapping. System 1 is dominated by sensory and especially visual thinking, while System 2 is dominated by verbal thinking.

System 1 makes most of our decisions,completely intuitively through simple rules of thumb (based on a lifetime of experience of course). System 2 does not have access to most of what System 1 is doing, and they speak in different languages. Unfortunately for market research, many standard approaches are only accessing System 2.

Context is consistently under investigated in market research. We are all prone to the Fundamental Attribution Error. That is we all over-emphasise the role of personal causes for the behaviours of others rather than the role of context and social situation. I say others because when it comes to explaining our own behaviour, attribution depends on whether the behaviour is considered good or bad.

Context has an important role in memory, providing clues to help us find the relevant memories of previous experiences. When two groups of deep-sea divers were given a memory test, the accuracy of their recall depended on them being in the same situation as when they were given a set of words to remember. For those who were given the words while under water, their recall was significantly higher when under water than when sat on the beach. And for those divers who were given the words on the beach, their recall was significantly higher when on the beach than when asked to remember under water.

I have myself seen this effect in research, for example when the richness and accuracy of brand tracking data for alcoholic drinks improved markedly by moving from street intercept interviews to ones conducted in bars.

And while researchers often considers the importance of context in ensuring the right order of questions, we too often miss the opportunity to frame individual questions in more meaningful ways. More importantly, order effects show us that asking other questions influences the answers we get in research.

This brings me to the rating problem. If you do research with two groups of consumers and get them to rate different jams for their taste, and for the second group you also ask them to give reasons for their preferences, then you get different answers. That is, the groups prefer different products. And you can guess which group’s preferences correspond most closely to market reality. Similarly, people who are asked to describe a wine after drinking it, are then worse at recognizing the same wine later.

The impact of asking questions has not been investigated as much as it should in research, but it has been studied rigorously in criminal psychology. And the studies show that if a witness is asked to describe a criminal, their ability to recognize the criminal afterwards is seriously compromised.

There are two reasons for this.  Firstly, our ability to recall memory is much less accurate than our ability to recognize something we have seen before. System 1 is more powerful than System 2 and the richness of our visual perception is much greater than our vocabulary. Secondly, memory is reconstructive, and each time we recall an experience, the experience changes as we update it with newer relevant experiences. Memories are not set in stone, but are constantly being updated.

Some of these studies have shown that the compromising effect of asking for descriptions can be minimized in three ways.

  1. Expert knowledge like that of wine tasters reduces the effects (as experts have richer vocabularies and conceptual frameworks to make sense of experiences)
  2. Being in the original context where the experience happened reduces the effects (which is why witnesses are asked to go back to the scene of the crime)
  3. Using free recall has much less impact than asking directed questions

What is to be done about these challenges to questions and answers? Here are three ideas for moving research beyond questions and answers.

Firstly, we can make much greater use of experimentation, including direct and indirect behavioural measurements and designed experiments. Direct measurements include actual purchase or other behaviours of interest (such as click through or visiting a store). Indirect measurements include biometric measurements such as facial expressions, physiological reactions and brain activity. However, many such measures are still at a stage where they are laboratory based and therefore lack contextual validity.

To give one example, in product testing it is possible to create stronger and more predictive models of behaviour by testing products with specific formulation changes and then modeling the data to build understanding of the drivers of consumer acceptance. Such approaches work best, when the product context is as specific as possible.

Indeed the approach of split testing is a powerful tool for optimizing websites, and similar approaches can be much more widely used if they are better understood.

A second answer is to move beyond words to use tools that dig into System 1 with images and metaphors, which use recognition more than recall and understand context through the observation of behaviours. There are many techniques that use non-verbal cues, including projective techniques that have a long history in research.

These tools are powerful, because our ability to recognize patterns is much greater than our ability to recall events. In fact, the brain is a supercharged pattern recognition machine, and the brain processes images hundred times faster than words, with all their richness of associations.

That is why in quantitative research, it is much better to rely on recognition rather than recall, for example by using selection tasks rather than rating tasks. Implicit testing is another approach which can help us understand what associations we have with a topic without asking directly.

“Power corrupts. PowerPoint corrupts absolutely.” – Edward Tufte

Questions and answers are to research what PowerPoint is to storytelling, They corrupt our understanding of what really drives behaviour.

None of us think in bullet points, or in pieces of information. We think, we experience and we remember in stories, and these need to be at the heart of consumer understanding. Stories are the most efficient way to learn, remember and share information, and reflect the brain’s need to search for patterns, and especially those patterns which link events in time (explaining cause and effect).

This is why cultural understanding, storytelling and archetypal frameworks can be powerful research tools, as well as powerful business tools.Customer stories are not just about product attributes or even the functional value of those attributes, but they are about emotional drives and ultimately life goals, and the jobs that consumers need to get done to achieve their goals. Simply put, stories are about characters, goals and challenges.

In summary, the way to get beyond questions and to answers that better reflect the real drivers of human behaviour, research must embrace:

  1. Experimentation linked to behavioural measurement
  2. Using fewer words to engage System 1
  3. Listening to and sharing real consumer stories

[This is a version of a presentation for the Asia Research Magazine’s Future Featuring seminar on 13th September 2012]

No responses yet

Leave a Reply