Multiple Mode Data Collection: Innovative Approaches 4 |
|
| Session Organisers |
Dr Scott Leary (Internal Revenue Service) Dr Jocelyn Newsome (Westat) Ms Brenda Schafer (Internal Revenue Service) |
| Time | Wednesday 17th July, 11:00 - 12:30 |
| Room | D16 |
Single-mode surveys -- whether telephone, web, or mail -- all have limitations that can result in coverage error. For example: telephone surveys may not reach households that have dropped landlines in favor of cell phones; web surveys are not an option for those without internet access (or those reluctant to click on survey links for fear of phishing scams); and mail surveys do not always make it to the intended recipient (and aren’t always opened or returned even when received).
In the face of these challenges, researchers are increasingly turning to mixed-mode surveys that afford the opportunity of combining multiple survey modes. Using multiple modes offer a variety of benefits to researchers. They can help improve response rates and reduce nonresponse error by improving coverage (Dillman, 2014). They can also speed up data collection and lower costs by allowing researchers to begin with a less expensive method of data collection and then move to a more expensive mode only for non-respondents (de Leeuw, 2005; Pierzchala, 2006; Tourangeau, Conrad, & Couper, 2013). Some researchers suggest that a sequential approach may be most effective, where a mail questionnaire is offered first and a web questionnaire offered later in the data collection process (Dillman, 2014) and others advocate for web first followed by other modes (Millar & Dillman, 2011), since the web is a very cost-efficient method.
However, as researchers embrace mixed-mode designs, it is not always clear how best to offer the various modes. Is it best to offer different modes concurrently or sequentially, and if sequentially, which mode works best if offered first? This session will explore innovative approaches used in multi-mode surveys. Researchers are invited to submit papers discussing experiments or pilot studies on any of the following topics:
• Multi-mode survey design experiments, including the use of smartphones and tablets;
• “Web push” experiments, including the impact on survey response and nonresponse;
• Discussion of differences in mode impact at different points in data collection; and
• Impact of mode sequencing on hard-to-reach populations.
Keywords: multimode, mixed mode, web push
Mr Douglas Williams (Westat) - Presenting Author
Dr Tala Fakhouri (National Center for Health Statistics)
Household screening is common when information about the characteristics of household members is used in selection. When key characteristics have a low prevalence, or are oversampled this can result in a large number of sampled households screened, but not selected. For face-to-face surveys this can be inefficient and costly, especially in an environment of declining response. A multimode approach using a mail push-to-web is an attractive alternative due to lower cost and high internet penetration. However, little is known about the comparable data quality properties between face-to-face and web modes. Face-to-face screening is considered a gold standard approach, yet respondents may fail to report household members when eligibility criteria are clear, or interviewers may unintentionally screen out reluctant respondents. For self-administered modes respondents sometimes fail to report unrelated household members, or young children. In this paper, we report on the results of an experiment comparing face-to-face screening and screening by web in the National Health and Nutrition Examination Survey (NHANES). This experiment is unique as households randomly selected for web screening were also again screened by an interviewer during an in-person visit. We report on the comparability of household characteristics between modes to determine if web screening provides data equivalent to face-to-face screening. We examine time between the web and face-to-face screening to see if true change can account for differences. In the presence of conflicting data we examine household selection criteria based on the screening data to see 1) false positives would negatively impact potential cost efficiencies, and 2) how false negatives would affect the initial sample needed for selection targets.
Dr Pablo Christmann (GESIS - Leibniz Institute for the Social Sciences ) - Presenting Author
Dr Tobias Gummer (GESIS - Leibniz Institute for the Social Sciences )
Mr Sascha Hähnel (GESIS - Leibniz Institute for the Social Sciences )
Professor Christof Wolf (GESIS - Leibniz Institute for the Social Sciences )
The ongoing trend of declining response rates and increasing costs challenges large-scale face-to-face data collection efforts in many countries. This development yields the practical question if it is possible to conduct General Social Survey-type studies in more cost-efficient self-administered modes. Most comparisons between self-administered and interviewer-administered surveys focus on assessments of survey quality indicators, such as representativeness, response rates, the occurrence of straightlining, or the number of refusals and “don’t knows” given by respondents. Yet, when facing a decision on whether to implement a survey by face-to-face, web, or mail, researchers are also interested in the possible effects of mode choice on the substantive conclusions they draw from their analysis.
Researchers only rarely have the opportunity to investigate and directly compare the impact of different survey modes on survey responses in large-scale probability-based surveys. The most recent wave of the European Values Study (EVS) in Germany provides such an opportunity to study these differences in an experimental design. Based on a register sample of the general population, respondents were randomly assigned to a face-to-face survey and a mixed-mode survey (web and mail). This experimental setup allows us to test whether survey responses vary across survey modes. For this purpose, we analyse all 138 agreement items of the EVS-questionnaire and compare differences between survey modes.
Overall, our analysis shows that the survey mode does not substantially affect survey responses when comparing means or standard deviations. Although we find statistically significant differences for about 33% of the items, the observed differences are very small and there appears to be no systematic directional effect. Results also hold when recoding the various scales into dichotomous agreement items and when regressing the agreement items on various socio-demographic factors and comparing predicted values instead.
Dr Rebecca Medway (American Institutes for Research) - Presenting Author
Mrs Danielle Battle (American Institutes for Research)
Mrs Rachel Hanson (American Institutes for Research)
Offering both paper and web modes of response is an attractive – and increasingly employed – design feature for maximizing response rates, minimizing bias, and controlling costs. However, there are challenges for employing such designs. While researchers tend to prefer web response, respondents may prefer paper response (Shih & Fan 2008). Offering both modes concurrently can reduce the response rate as a result of the “paradox of choice” (Medway & Fulton 2012), but offering them sequentially may prematurely dissuade sample members from responding who would have done so had they known the second mode was going to be an option later. Recently, Biemer et al. (2017) reported promising results of a “choice-plus” protocol that addresses some of these challenges; it presents both modes concurrently but incentivizes web response by offering a larger contingent incentive for web response than for paper response.
This presentation will report the early results of an experiment that builds on these findings. Households sampled for the 2019 National Household Education Survey (NHES) will be randomly assigned to receive (1) a paper-only protocol (“paper”), (2) a sequential web-paper protocol (“web-push”), or (3) a choice-plus protocol. Key differences from the Biemer et al. study include: a contingent incentive will be offered only for web responses (not for paper) in the choice-plus condition; a contingent incentive will not be offered at all in the other conditions; and the NHES will use a two-phase design. We will compare the response rate by condition. Next, we will evaluate the effect on the representation of hard-to-reach households. Finally, we will determine whether the choice-plus condition leads households to respond more quickly or increases the percentage of responses received via web. This presentation will be of interest for practitioners interested in mixed-mode survey design – and whether different protocols are more effective for hard-to-reach populations.
Dr Heinz Leitgöb (University of Eichstätt-Ingolstadt) - Presenting Author
Mrs Nathalie Leitgöb-Guzy (Federal Criminal Police Office)
In their seminal book on survey designs, Dillman et al. (2014, 427ff) advocate the application of sequential mixed-mode (MM) designs that initially avoid offering a simultaneous choice of response modes. Besides theoretical considerations about the decision-making process in a mode-choice decision, the recommendation is based on empirical evidence from two studies: (i) a meta-analysis comparing a total of 19 simultaneous MM (mail & web) and mail-only surveys and (ii) a survey experiment conducted among U.S. college undergraduates. However, inferences drawn from the results of these studies may not be valid for European countries, as they focus almost exclusively on North America.
For Germany, relevant studies are rare, but existing results exhibit a tendency in favor of simultaneous MM designs. To contribute further evidence particularly for mail & web MM designs, we conducted a survey experiment in a large German city by randomly assigning (i) a simultaneous and (ii) a sequential (web first) MM survey to a total of 2,000 individuals drawn from the resident population via simple random sampling. Following the total survey error perspective, we introduce various representation (e. g. unit nonresponse, divergence between known population and sample distributions) and measurement (e. g. item nonresponse, prevalences in sensitive items, measurement error) indicators to ensure an exhaustive comparison. In accordance with existing German evidence, first results indicate higher response rates and overall data quality in the simultaneous MM design.
References
Dillman, D. A., Smyth, J. D. & Christian, L. M. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys. The Tailored Design Method. Hoboken: Wiley.