Phrasing the Question

From Market Research
Jump to: navigation, search

Writing a question involves selecting a question type and phrasing the question. For the novice researcher, the secret to wording questions well is to copy others’ work. Plagiarism is at the heart of asking good questions, as there are many, many ways of wording a question poorly. The easiest way to get across the basics is to look at a few examples of bad questions (all from real studies): What is your principal brand of soft-drink? What problems can you see with this question? 'Principal' is an overly technical word. Sure you know what it means, but what about the guy who collects the garbage? A better word than principal is 'main'. However, this does not remove the ambiguity from the question: is it asking which one is liked the most, or, bought the most often? Such a question will likely end up measuring brand salience (i.e., which brands come to mind) rather than anything else.[1]

And another example:

Do you prefer…  
[] Beer  
[] Wine  
[] Spirits  
[] Non-alcohol drinks

There are lots of problems with this question. What about cider drinkers? They are ignored (i.e., the categories are not exhaustive). Furthermore, people that do drink are likely to drink different types of alcohol at different times, such as beer after sport, wine with dinner and spirits after dinner; how should such a person answer the question?

Contents

Key principles when phrasing a question

Principle 1: The question must be answerable

The least-educated dumb person that is likely to have to answer the question needs to be able to figure out how to answer the questions accurately.

Principle 2: The question must not be ambiguous

The issue is not one of respondent confusion, as this is already addressed by the first principle and respondents are remarkably adept at answering incomprehensible questions (presumably because this is the only way they can finish many questionnaires). The problem with ambiguity is to avoid situations where, once we have collected the data, we cannot discern what it means. For example, one study asked flyers about the importance of getting 'discounts' and found that it was important. However the research did not clarify if the flyers were happy with a 2% or 20% discount, the research was ambiguous and could not be used to derive valid insights. To avoid being ambiguous, questions need to focus on the current, the specific and the real.[2] A trick to achieving this is to focus on writing questions around the 5Ws – who, what, where, when and why.[3]

A good way of checking for ambiguity of questions is to use thinkaloud interviews, in which real respondents are asked to answer the question but are required to verbalize all their thoughts while answering the question. A shortcut to this can be to get the dumbest person you know to complete the questionnaire and comparing their answers with what you believe to be true.

Principle 3: Incentive compatibility

The third principle is that questions must be incentive compatible, which is a term of art in economics, and it means that questions need to be written in a way that people have an incentive to provide honest data. Consider the question:

Are you aged under 35? [] Yes [] No

If asked as a screener at the very beginning of a questionnaire, respondents conclude that an answer of No will cause them to be screened out of the study (i.e., asked no more questions). Consequently, when this question is asked at the beginning of questionnaires where people are being paid more if they complete the whole questionnaire, some respondents lie and pretend they are aged less than 35, when they are actually older. By contrast, if asked at the beginning of a questionnaire where people are not being paid to do the study, people aged under 35 pretend to be older, as this becomes a polite way of refusing to participate. An incentive compatible way of screening people based on being aged under 35 instead asks:

How old are you?
[] Under 18 
[] 18 to 24 
[] 25 to 34 
[] 35 to 44 
[] 45 to 54 
[] 55 to 64 
[] 65 or more

and then screens people out if they select one of the older age groups. It is “incentive compatible” because there is no incentive to answer the question dishonestly.

Imagine yourself charged with the problem of pricing an iPad before they were launched. How much should you charge? There were no competitors, and thus no straightforward way of working out even a ballpark price point. A simple approach to this problem is to ask people how much they might pay: What is the most you would pay for the product?.

The amount that people nominate is referred to as either the reservation price or their willingness-to-pay for the product. The analysis then proceeds by assuming that people will buy a product if its price is less than or equal to the amount nominated by the respondents.

Such questions are not not incentive compatible. Imagine that you loved the concept and would be prepared to pay $2,000 for an iPad. Would you tell this to an interviewer? If you did, it could result in the product being sold at $2,000. Whereas if you lied and said $1,000, you may end up with the product being sold for less and you not having to spend so much. Sure some respondents may be honest, either because they are by nature honest or because they believe that by being honest they will maximize their chance of the product they wish to buy being sold, but others may lie. Observing that respondents have an incentive to lie does not prove that they will lie (unless one is an economist). However, it is better to figure out ways to conduct research that are incentive compatible, as then the risk is removed (this is discussed in Advanced Questions and Questionnaires).

Frequency questions

Perhaps the most frequent mistakes made by novice researchers relate to frequency questions. Consider this question:

How often do you go to the cinema?
[] Never  [] Rarely  [] Sometimes  [] Often

What do the scale points of Rarely, Sometimes and Often mean? One person’s often may be another person’s sometimes. When measuring quantities it is always important to give people precision about time and frequency, such as:

Thinking about the last twelve months, how many times did you go to the movies?
[] None 
[] Once or twice  
[] 3 to 6 times 
[] 7 or more times

A problem with such a question is that respondents may not be able to answer it accurately because they will not remember the precise number of trips to the cinema from the past twelve months. Such a reservation is justified. A solution is to reduce the time interval:

Thinking about the last two weeks, that is, everything from today back to Wednesday two weeks ago, how many times did you go to the movies?
[] None  
[] Once  
[] Twice  
[] 3 to 6 times 
[] 7 or more times

Asking about such a short time period creates a different problem. Attendance at the cinema is sporadic for most people. There is a big difference between somebody who goes six times a year and somebody who does not go at all, but asking about the last two weeks can result in such people being grouped together and, thus, it is better to ask about the longer time interval.

So, what then is the best way to ask about frequency of going to the cinema? A question like:

Thinking about the last twelve months, how many times did you go to the movies?
[] None  
[] Once or twice  
[] 3 to 6 times 
[] 7 or more times

perhaps remains the best approach, as even though many people will not be able to answer with complete accuracy, most of the people who have not gone will be able to select None and most that have gone more than seven times will be able to select 7 or more times, whereas with the vaguer Rarely, Sometimes and Often categories, we can not make any deductions about what they actually mean.

Another example:

How often do you visit the web pages of this site?
[] Several times a day
[] Once a day
[] Once a week
[] Once a month
[] Less often than once a month
[] This is the first time I have visited

These categories are not mutually exclusive (anybody who can tick the last category can also tick the second last category). And, they are not exhaustive. What should somebody answer if they visit two or three times a week?

Next page

Ordering the Questions in the Questionnaire

References

  1. Sudman, Seymour and Norman M. Bradburn (1987), Asking questions: A practical Guide to Questionnaire Design. London: Jossey-Bass.
  2. Converse, Jean M. and Stanley Presser (1986), Survey Questions: Handcrafting The Standardized Questionnaire. Beverly Hills: Sage Publications.
  3. Sudman, Seymour and Norman M. Bradburn (1987), Asking questions: A practical Guide to Questionnaire Design. London: Jossey-Bass.
Personal tools
Namespaces

Variants
Actions
Navigation
Categories
Toolbox