Deciding How to Conduct the Interviews
The vast majority of quantitative market research surveys are conducted online. Nevertheless, there are many other ways of data collection, including in-store interviews, phone interviews, online qual, SMS interviews, door-to-door interviews and mail interviews.
There are numerous different considerations that need to be traded off when choosing how to collect data, including cost, time taken to conduct the research, the ability to show stimulus (e.g., videos, images). In the main, these trade-offs are straightforward to make. The only difficult issue in most real-world surveys relates to coverage.
Different ways of conducting interviews 'cover' different proportions of the population. For example, if you work out what the market works by asking your best friend, you have only covered a minuscule proportion of the market. The proportion of the market that can potentially be included in your survey is its coverage. For example, if you decided to conduct a survey by phoning 200 people on a Wednesday evening, the coverage will consist of all the people who:
- Were at home at the times that you could have called them.
- Will have answered the phone.
- Were in the list of phone numbers that you selected your sample from.
Note that the idea of coverage relates to people that you could potentially have contacted, as opposed to those that you did contact.
Where the people in the population that are not able to be contacted by your way of collecting the data are different in important ways from those that can be contacted then the survey will suffer from coverage error.
All surveys suffer from some degree of coverage error. The problem is that we usually have no easy way of working out the likely coverage error and need to apply commonsense.
Minimizing coverage error
It is good practice to choose to collect data in the way that will reduce coverage error. For example, many service companies still conduct important market research by phone, even though it costs more than ten times as much per interview as an online interview. They do this because they often have their customers’ phone numbers but are less likely to have all off their customers email addresses (i.e., the list of email addresses has worse coverage than the list of phone numbers).
Coverage error is an endemic problem in market research. As a general rule, the cheapest forms of data collection suffer from the worst sample selection biases. Consider the phone in polls used by TV stations and the web polls run by online newspapers.
Consider a chain of gyms wanting to understand how to increase gym attendance. It seems evident that the people that have the most active lifestyles will be spending less time using computers and are thus less likely to participate in online surveys, meaning that if an online survey is conducted it will over-represent unfit people and thus using an online survey will result in coverage error'.
While it is clear that conducting an online survey is far from the ideal way of conducting a survey of looking at the exercise market, all of the other options suffer from similar problems and some, such as phone and door-to-door interviewing are considerably more expensive, and thus, unless hundreds of thousands of dollars are available for the survey, conducting such a survey online is usually the only practical solution. Importantly, this does not make the online sample 'valid'. Rather, it simply makes it the tallest dwarf and we need to keep this in mind when interpreting the data from such a survey (and from most online surveys for that matter).