Asking Questions Without Asking The Question

Apr 20 2012

Curiosity has it’s own reasons (Einstein)

The lifeblood of market research is curiosity and curiosity is a great thing in all aspects of life (as Einstein said so eloquently on several occasions). Market researchers are very adept (and trained) to ask lots of questions, but I think we ask far too many and should ask far fewer and be smarter in the way we design research in order to do that. Let me be clear from the start. Asking questions in market research is very often at best a waste of time, and at worst positively misleading.

I have three main concerns with asking questions which I would like to share before talking about some alternative strategies that have been used in product testing for some time and I think should and will be increasingly used in market research in the future.

It’s very clear that market researchers ask too many questions. Questionnaires are too long, and one of the barriers to moving research to mobile is the challenge of ‘moving’ current surveys to a medium where attention span is short and screen size is small. This will have to change and interesting that Google’s survey product is based around a model of asking one or two questions only (more on this later).

The elephant in the room (and his rider)

Most importantly, recent discoveries in behavioral economics, neuroscience, and psychology highlight what I think many of us have always known. Just because you can ask a question, it does not mean that the person you ask knows the answer or can give you an accurate response. I’m not talking about lying or social bias here, but rather our mental capacity as humans to understand and articulate the reasons behind our behaviours.

Thus, the first concern with asking questions is the importance of unconscious or implicit knowledge. Daniel Kahneman’s explanation of System 1 and System 2 processing makes it clear that most of the things that make us ‘clever’ are things outside of our awareness and control. I would like to focus on one aspect of this difference between the elephant and the driver which is very relevant in sensory testing.

If any of you have taken tests of selective attention, you will realize that we often miss much of what’s happening around us. Or rather we don’t miss things, our brain ignores them, in order to focus on what is immediately important to achieving our goals. That’s why we don’t “see” the gorilla. In fact our eyes do see the gorilla, although for many of us our brain ignores it because we are focused on a specific task.  This is what is called attentional blindness. Eye tracking of participants in such tests show that there are no differences in the eye’s attention to events in those who spot the gorilla and those who do not. The differences are hidden deep in the brain.

The reason our brain ignores much information is quite simple. By some (well researched) estimates, our brain takes in more than 11 million pieces of sensory information each second of each day  (the majority of which is visual information). Our conscious (reflective) brain can handle around 16 pieces of information each second (40-50 if you take the highest estimates). That is, we can actually focus our conscious attention on slightly more than one millionth of the information available to us.

This means that our brain has to filter out most of the information it receives, processing it through our unconscious brain. That doesn’t mean that this information is all ignored.  It can impact our decision-making, but we are not conscious of that impact. And that is why most of the time we are not really aware of what motivates and drives our behaviors. Or in the language of product tests, that is why, ‘we know what we like but we don’t know why’.

This leads to the second challenge, which is the importance of framing effects and implicit priming. Our memory doesn’t work like a filing cabinet, but by association and by linking pieces of information together. If you think about the sensations of enjoying a product or any kind of brand experience, in reality each part of that experience is filed in a different part of the brain (taste, touch, smell, etc). The central link between those different perceptions is the emotional experience. To access such memories (which we do all the time to predict the future), the brain uses mental shortcuts (sometimes called heuristics) to access the relevant information.

Think of shopping in the supermarket. We use proximity to infer categories, packaging to infer product job, and color to infer brand and other cues. These are mental shortcuts we use to make life simpler. You’ll see that one theme of behavioral economics is that our brain is wired to simplify life as much as possible (otherwise it would be overwhelmed with decision making, and would not have the energy supplies to meet demands).

Context, context

That’s why context is so important to research although not the main focus of this article. Context gives our brains the cues to find the right information to answer the question. These cues are rarely the same ones that client companies use to categorize their brands and products. There is often a tendency to ask tracking questions by referring to retail audit categories rather than the categories in consumer minds (for example, the jobs they perform in consumer lives). One of my colleagues a few years ago made some changes to a brand tracking study of alcohol products, and one of the changes he made was to move the interviews into bars where the context gave the relevant cues to participants. The accuracy of response improved dramatically.

These effects are why it is important to consider the order in which we ask questions in research, and how we word those questions. We all know that order effects exist, but the inevitable and logical conclusion of this is not just that we should get the order right.

That such effects are real means that asking a question of itself can change the participant’s reality.

Lurking in the shadows

This leads me to my third and final point. Asking questions changes the consumer’s experience and therefore changes the responses they give. It changes the frame of reference that is used to evaluate something. There is no such thing as an unbiased (or unbiasing) question.

This effect is best documented in criminal psychology. It is known as verbal overshadowing, and has been well documented (first discovered by the Schoolers). The effect is simply that when we talk about something we’ve experienced (faces or colors for example), then we lose our ability to accurately remember the experience.

So being asked to describe something, or being asked questions about it, distorts our memory and sometimes blinds us to the experience. This may be because of the need to take mental processing power in order to use words and descriptions (language takes up a lot of conscious processing) and also because memory is reconstructive (that is, we mentally simulate an experience when we reacll). And each time we reconstruct a memory, it changes.

Lining up memories

Most dramatically, when a witness is asked verbally describe a perpetrator, this makes them less likely to identify the real criminal in a line up than someone who is not asked (the differences are sometimes dramatic). These effects have also been demonstrated in sensory testing, for example on wine.

Applying words compromises our ability to accurately recall an experience. In sensory research, a picture (non-verbal) is a better way to remember a stimulus.

So our recognition is much more accurate than our recall, something that system designers have known for a long time. The effects can be partly mitigated by three things.

  1. Recall is better in the same context as the original experience (which is why getting someone back to the scene of the crime helps memory)
  2. Free recall questions have much less negative impact than elaborate and directed questions (allowing the brain more freedom to apply it’s own framing)
  3. The more someone knows about the topic, the less compromised they are (because they have a more elaborate mental framework and language to make sense of the experience). Expert wine tasters are impacted much less than naïve consumers when asked to describe a wine they have tasted.

Go to the experts

The last point is particularly relevant for product testing. In many sensory tests we want to understand drivers of liking, acceptance, purchase, satisfaction or whatever our dependent variable might be. It could also be consumption or a happy face too (non-verbal reactions are important).

In order to understand drivers of product liking, we need to know two things. Firstly, the consumer’s overall response. Secondly, the differences between the products. It’s not necessary for consumers to do anything more than give an overall response to the products, as long as we can describe them (or more importantly describe the differences between them). This can be done using trained sensory panels (who are experts and therefore less compromised), or through more physical descriptions of the products combined with appropriate experimental designs and analyses which allow us to understand the drivers of liking without asking consumers about them. See Malcolm Gladwell’s TED talk on spaghetti sauce to hear more about this (or read this article). These approaches have never been used as much as they could or should be, as they can provide long term strategic direction to product development as well as short term tactical guidance.

Using design instead of questions

There is no better time than now to start using these approaches more. In the era of big data and digital research, work such as website optimization is using exactly these kinds of tools. What companies like Autonomy do for their clients (and what Amazon do for themselves) is to manipulate website layout and design variables and measure behavior. The behaviours they measure are often very different from the behaviours our ‘conscious’ (System 2) brain might expect, because if has very different priorities to our unconscious (System 1) brain. Our attitudes and opinions do not match how we really behave online and elsewhere.

Google Surveys may offer the opportunity to many more clients to do this, offering easy access to simple ‘split testing’ using one or two key questions (something which will be more and more important as research goes mobile). These surveys can then be combined with experimental testing and, powerfully, with all the other data which Google has access to about the behaviours and demographics of their online population. Potentially a very powerful combination.

So here are three ways to avoid the pitfalls of asking questions:

  1. If you can, measure real behaviours instead of opinions, attitudes and beliefs. Measure purchase, click through, consumption, non-verbal responses (facial expressions for example).
  2. If you have to ask questions, focus on the one or two most important questions you need to ask (behavioral responses).
  3. Use survey design, and data from other sources, to help you interpret the drivers of response and give you the answers you need more accurately.

This is an edited version of a talk prepared for’s BE event on 19 April 2012. A recording of this is available at

No responses yet

Leave a Reply