There are variety of ways to collect survey data, including phone, Web and mail, each with pros and cons that can ultimately impact the quality of the data gathered – for better and worse.
There are variety of ways to collect survey data, including phone, Web and mail, each with pros and cons that can ultimately impact the quality of the data gathered – for better and worse. To better understand how the survey mode affects responses related to real estate behaviors, Zillow conducted an experimental study in March 2017 that administered the same questionnaire to a sample of 2,016 adults via telephone and 2,011 via Web.
The Web survey was self-administered by the online panel respondents themselves. The phone survey was administered by a trained interviewer using a computer assisted telephone interviewing system (CATI). Random digit dialing (RDD) was also used in the phone survey to ensure an equal probability of selection into two independent probability samples, a cell phone and landline sample. Both the phone and web survey were weighted to balance the sample to reflect the U.S. population of adults 18 and older based on the same set of demographic attributes – age, sex, geographic region, race and education.
General trends identified in each survey were broadly consistent. Mode differences in the responses between the two surveys was common, but they were typically small. Substantive differences emerged on questions involving both respondents’ use of technology and on more personal topics.
For example, in the online survey 65 percent of renters who moved into their home in the past year said they used online resources such as websites and apps. Only 59 percent said the same in the telephone survey. In addition, the telephone survey respondents said they were more likely to reach out to people during their search – landlords, friends, relatives or neighbors – than the Web survey respondents.
This difference makes sense: To respond to a Web survey, one needs to access… the Web. Phone respondents used the internet less frequently – only 89 percent of the telephone survey respondents said they connected to the internet in the past 12 months – and accessed it on fewer device types. Clearly, to achieve accurate responses, it’s especially important to adjust for internet use in the population during the weighting process.
In terms of who responds to online and phone surveys, we found that telephone respondents are more likely to both live in rural areas and lean Republican than Web survey participants – consistent with other studies.[1]
In addition to who responds, there are differences in how people respond. This is especially true for more sensitive topics prone to social desirability bias. Survey respondents may not feel comfortable sharing personal information with a live interviewer and/or may want to be seen positively by that person. For example, respondents may be less likely to report drug and alcohol use to an interviewer.[2]
This phenomenon was apparent in our phone survey when respondents were asked about their household income: One-in-five phone respondents refused to report their earnings. Respondents at the lower and higher end of the income distribution are less likely to respond, and this can potentially alter results.[3]
A web survey can be perceived as more anonymous, and so be more insulated from social desirability bias. Median household income was reported as $55,000 in the Web survey, approximately $10,000 higher than what was reported in the phone survey,[4] and closer to benchmark 2016 household income reported by the Census ($57,617).[5]
While Web surveys may be more appropriate at capturing accurate information on questions prone to social desirability bias, phone surveys usually have less item-level missing data (when respondents skip survey questions) and can be less prone to some types of satisficing, or shortcuts respondents do such as straight-lining responses down a list of questions. Trained interviewers can also help respondents understand more complex survey questions in a way that’s usually impossible in a Web survey.[6]
Mixed-mode surveys can be a useful approach for studies when a probability sample is desired, and it’s not too expensive to do initial recruitment and pre-screening for eligibility on the phone. The remaining data can be gathered online or through a mail survey. This allows for a probability sample to be constructed with most information obtained in a mode less prone to social desirability bias.
However, collecting data via probability phone samples can be infeasible for groups with a low-incidence in the population (e.g. people who recently sold their home). Interviewers must pre-screen a very large number of calls before finding eligible respondents, which can be done more efficiently using online panels. In addition, response rates to telephone surveys have dwindled significantly over recent decades, which limits their viability as true probability samples. This is a trend that is likely to continue[7] prompting many in the industry to switch entirely to non-probability web surveys.
By far the largest advantages of Web surveys over phone surveys are that they are much less expensive and allow for faster data collection, and we should continue to understand how to collect high-quality data from Web surveys. But online non-probability samples present many challenges, and the quality of Web surveys can vary dramatically depending on the methods used.
Further research details some of the choices Zillow has made in designing Web surveys.
[1] Pew Research Center, May 2015, “From Telephone to the Web: The Challenge of Mode of Interview Effects in Public Opinion Polls”.
[2] Tourangeau, Roger and Ting Yan. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin.
[3] Bollinger, C.R., Hirsch, B.T., Hokayem, C.M. and Ziliak, J.P., 2014. Trouble in the Tails: Earnings Non-Response and Response Bias across the Distribution. In Annual Meeting of the Society of Labor Economists, May (pp. 2-3).
[4] Prior outside research shows that median incomes estimated from phone surveys can also be higher than estimates from web surveys. Both ends of the distribution suffer from social desirability bias. Pew Research Center, May 2015, “From Telephone to the Web: The Challenge of Mode of Interview Effects in Public Opinion Polls”.
[5] Guzman, G.G., 2017. Household Income: 2016. American Community Survey Briefs. United States Census Bureau.
[6] Lavrakas, P.J., Benson, G., Blumberg, S., Buskirk, T., Cervantes, I.F., Christian, L., Dutwin, D., Fahimi, M., Fienberg, H. and Guterbock, T., 2017. The Future of US General Population Telephone Survey Research.
[7] Lavrakas, P.J., Benson, G., Blumberg, S., Buskirk, T., Cervantes, I.F., Christian, L., Dutwin, D., Fahimi, M., Fienberg, H. and Guterbock, T., 2017. The Future of US General Population Telephone Survey Research.
