Multiple Mode Data Collection: Innovative Approaches 2 |
|
| Session Organisers |
Dr Scott Leary (Internal Revenue Service) Dr Jocelyn Newsome (Westat) Ms Brenda Schafer (Internal Revenue Service) |
| Time | Tuesday 16th July, 14:00 - 15:30 |
| Room | D16 |
Single-mode surveys -- whether telephone, web, or mail -- all have limitations that can result in coverage error. For example: telephone surveys may not reach households that have dropped landlines in favor of cell phones; web surveys are not an option for those without internet access (or those reluctant to click on survey links for fear of phishing scams); and mail surveys do not always make it to the intended recipient (and aren’t always opened or returned even when received).
In the face of these challenges, researchers are increasingly turning to mixed-mode surveys that afford the opportunity of combining multiple survey modes. Using multiple modes offer a variety of benefits to researchers. They can help improve response rates and reduce nonresponse error by improving coverage (Dillman, 2014). They can also speed up data collection and lower costs by allowing researchers to begin with a less expensive method of data collection and then move to a more expensive mode only for non-respondents (de Leeuw, 2005; Pierzchala, 2006; Tourangeau, Conrad, & Couper, 2013). Some researchers suggest that a sequential approach may be most effective, where a mail questionnaire is offered first and a web questionnaire offered later in the data collection process (Dillman, 2014) and others advocate for web first followed by other modes (Millar & Dillman, 2011), since the web is a very cost-efficient method.
However, as researchers embrace mixed-mode designs, it is not always clear how best to offer the various modes. Is it best to offer different modes concurrently or sequentially, and if sequentially, which mode works best if offered first? This session will explore innovative approaches used in multi-mode surveys. Researchers are invited to submit papers discussing experiments or pilot studies on any of the following topics:
• Multi-mode survey design experiments, including the use of smartphones and tablets;
• “Web push” experiments, including the impact on survey response and nonresponse;
• Discussion of differences in mode impact at different points in data collection; and
• Impact of mode sequencing on hard-to-reach populations.
Keywords: multimode, mixed mode, web push
Dr Peter Lugtig (Utrecht University) - Presenting Author
Dr vera Toepoel (Utrecht University)
Dr Tom Emery (NIDI - Netherlands Interdisciplinary Demographic Institute)
Dr Susana Cabaco (NIDI - NIDI - Netherlands Interdisciplinary Demographic Institute)
Dr Detlev Lueck (Federal Institute for Population Research (BiB))
Miss Almut Schumann (Federal Institute for Population Research (BiB))
Dr Robert Naderi (Federal Institute for Population Research (BiB))
Mr Martin Bujard (Federal Institute for Population Research (BiB))
The paper presents results from a push-to-web experiment in the context of the Generations and Gender Project, which is a cross-national panel survey. In the past, GGS has been conducted in CAPI mode only.
In 2018, an experimental pilot study was conducted in three countries (Germany, Croatia, Portugal) to study the possibilities of implementing a push-to-web design. Respondents are first contacted by mail, and asked to participate in a web survey. After two reminders, nonrespondents are followed-up by an interviewer who will conduct a CAPI interview.
The experiment centered on contrasting the push-to-web design, to the old, CAPI-only format. In this presentation we show how the push-to-web design leads to substantial higher response rates than the CAPI-only design in two of the three countries in the experiment. We also discuss differences in the socio-demographic experiments between the groups to study differences in nonresponse between the push-to-web and CAPI only design, and shortly show that the incentive amount, timing and number of reminders and within-household selection procedure matter.
We further discuss the results for 1 country where the push-to-web design was less successful, and finalize by outlining what are success-factors for implementing a push-to-web design in a cross-national (European) context.
Dr Jocelyn Newsome (Westat) - Presenting Author
Dr Kerry Levin (Westat)
Ms Jennifer McNulty (Westat)
Dr Hanyu Sun (Westat)
Ms Brenda Schafer (United States Internal Revenue Service)
Mr Patrick Langetieg (United States Internal Revenue Service)
Mr Rizwan Javaid (United States Internal Revenue Service)
Dr Scott Leary (United States Internal Revenue Service)
The use of multiple modes of data collection has become standard practice in survey research. Multi-mode surveys can help reduce certain types of survey error, speed up data collection, and lower costs (Dillman et al., 2014). Within the multi-mode environment, researchers seek ways to encourage response via web, since it can speed response and result in significant cost savings.
However, “pushing” respondents to the web can be challenging. In this paper, we will discuss our experience with the U.S. IRS Individual Taxpayer Burden (ITB) Survey, a national, annual multi-mode survey that measures the time and money taxpayers spend complying with tax law. Although the ITB Survey is offered in multiple modes, the majority of respondents choose to complete a paper questionnaire. Since 2010, we have conducted multiple experiments to try to increase response via the web. These attempts have had a limited impact on web response (Newsome et al., 2012, Levin et al., 2013, Levin et al., 2015).
In the most recent fielding, we conducted an experiment to see if a web push would be more effective with respondents with international addresses. In the past, we have found that international respondents are twice as likely to complete the survey via web compared to those with domestic addresses. We have also found, based on feedback from respondents, that international mailings are more likely to experience delays and misdirected delivery.
For this experiment, half of the sample were asked to complete the survey via the web. The other half received a paper questionnaire, with web instructions on the back cover. We will examine the impact of the web-push on overall response rates as well as response via web. We will also examine any difference in response rates across demographic variables, including age, income, and geography.
Mr Todd Hughes (UCLA Center for Health Policy Research) - Presenting Author
Mr Brian Wells (UCLA Center for Health Policy Research)
Mr Royce Park (UCLA Center for Health Policy Research)
Dr David Dutwin (SSRS)
Concerns with decreasing productivity and increasing survey costs are leaving many surveys to consider abandoning their historic use of random-digit-dialing (RDD) and telephone interviewing (CATI) for the seemingly greener pastures of address-based sampling (ABS) and mixed-mode data collection including web. But such drastic design changes must be carefully considered when studies involve lengthy interviews, multiple survey languages, diverse target populations, and several interviews within a single household. Design decisions on contact strategies and supplemental sampling frames must be evaluated for efficiency and impact.
The California Health Interview Survey (CHIS) is the largest state health survey in the United States interviewing 20000 households a year from a population of over 11 million households (40 million individuals) where nearly 16% have limited English proficiency. Since 2001, CHIS has been an RDD/CATI survey conducted in six languages, but given the current survey climate, we are considering the need for a mixed-mode redesign.
This paper presents results from a CHIS field experiment using an ABS sequential mixed-mode design: 28000 households received a mail invitation push-to-web, with CATI follow-up. The experiment included multiple sampling frames in an effort to better reflect the diverse population groups in the state, and used different contact strategies and materials to obtain completes from ethnic and language minorities. We examine response rates, cost differences, and shifts in key estimates compared to CHIS production data. In general, the revised design makes large strides to improve response rates and reduce costs per complete for CHIS. The different modes and sampling frames did help to better include different population groups, including some groups under-represented in the historic RDD/CATI design, however, there are numerous shifts in key estimates that could disrupt historic trends, and some challenges remain for representing racial/ethnic minorities.
Mr Gijs van Houten (Eurofound) - Presenting Author
Mr Giovanni Russo (Cedefop)
Ms Femke Dekeulenaer (Ipsos)
Ms Ahu Alanya (Ipsos)
Mr Andrew Cleary (Ipsos)
The European Company Survey 2019 uses a push to web survey design, with telephone initial contact and short CATI screener questionnaire, and web interviewing for the full substantive sample. The three previous iterations of the survey have been conducted via CATI. The survey is undertaken by Eurofound, with this 4th edition taking place in partnership with Cedefop and fieldwork delivery commissioned to Ipsos. The survey informant is the manager responsible for human resources in a randomly sampled establishment and when possible an employee representative. Main survey fieldwork will take place in the first half of 2019, with over 20,000 interviews across 28 European countries, following a full pilot late 2018.
Our presentation will consider two aspects of the design. First, in the pilot we experimented with the informant targeted for the telephone screener questionnaire, with two conditions: a senior human resources manager, who would also be asked to complete the web interview, or any informant able to provide sufficient details about the establishment to complete the CATI screener. With this experiment we hypothesised that targeting a less senior informant would result in a higher response rate to the CATI screener but lower to the web interview. Second, we included additional questions in the CATI screener so that we could consider non-response bias in the web interview sample. For both aspects we will examine response rates and non-response bias, in relation to the experiment and overall, and across countries. We will also describe the design of the survey, including the contact strategy used to boost response and minimise non-response bias in the web sample, drawing conclusions relevant to others interested in conducting push to web business surveys.