Who Will Question the Questioners?

“Don’t ask silly questions if you don’t want foolish answers.”  - C. Ryland

I have a lot of sympathy with Roger Sant’s recent article in research-live.com on the need for researchers to provide more engaging and participative questions, and his recommendation to allow participants to define for themselves what is important.My own view is that this can be done in many ways.  With the use of text mining software, greater use of open end questions, can provide clear and detailed feedback on what customers value and what they ignore.  I would urge the use of intelligent open questions though, as “what did you like and dislike?” is rarely the best question to ask.  However, there is no reason not to use participant selection in quantitative surveys by, for instance, asking respondents to pick the three most important (or most positive) characteristics or image attributes for them.  In my experience, such questions often provide much greater differentiation and clarity in understanding the drivers of choice and behaviour.  They also make the tasks simpler and more engaging, as participants have to actively process options and make choices, rather than giving checkbox responses.

Questionnaire structure and content, along with question scales, are often topics of heated debate in research circles.  In the latest issue of International Journal of Market Research, Petra Lietz reviews well over 100 studies and summarises some key learnings.  The article should be essential reading for researchers who wish to make evidence-based choices for more effective questionnaire design.

Some of the key findings, which should be unsurprising, are:

• keep questions (and questionnaires) as short as possible
• introductory explanations to groups of questions can increase data quality
• make question terms as specific as possible, and avoid using vague words (eg ‘probably’)
• avoid hypothetical questions concerning future behaviours, and use vignettes and alternative scenarios when asking about future behaviours
• do not ask participants to recall events beyond a week/month, unless they are very important life experiences (ie ask about hospitalisation in the past year, but brand usage in the past week)
• keep language simple and clear (participants are more likely to say don’t know or to provide socially desirable answers when confused)
• indirect questioning can also overcome the propensity to give desirable answers, especially for sensitive topics
• use precise numeric reference points and time periods and avoid adverbs such as ‘frequently’, ‘possible’, etc

Petra Lietz also discusses scale use, and concludes that in general, 5 and 7 point scales are fine for most absolute judgements, although longer scales may help when more abstract judgements are required.  My own experience is that scales rarely make a difference to the decisions that are, or can be, made from data.  While longer scales can give greater differentiation, through greater variability, they rarely if ever give different answers to the business question.  Or to put another way, shorter scales provide the same information in most cases, and are easier for research participants to use (and for interviewers to administer).

The full article is worth reading, and I would love to see some published comparisons of using standard scales versus participant selection (I have only seen commercially sensitive ones to date).

Roger Sant is right to call for researchers to give participants a more active role in defining what is important and relevant for them.  This makes research more engaging, more relevant, more open, more trusting and I believe would also give clearer answers to clients’ questions.

REFERENCES

http://www.research-live.com/4003484.article Let’s let respondents decide what matters by Roger Sant

Research into questionnaire design: a summary of the literature by Petra Lietz, International Journal of Market Research, 52, 2, 2010, 249-272