All time references are in CEST
Innovative Uses of Multimode and Multidata in Surveys: Challenges, Success, and Error Trade-offs |
|
| Session Organisers | Dr Ting Yan (NORC at the University of Chicago) Dr Leah Christian (NORC at the University of Chicago) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Declining response rates and increasing cost of data collection continue to challenge survey practitioners, survey researchers, and government agencies. At the same time, there is a growing need for more data and for more data to be collected more frequently, timely, and cost effectively. As a result, surveys are increasingly employing multiple modes both to contact sampled persons and to collect data from them. For instance, sampled persons may be mailed postal letters and also visited by field interviewers inviting them to participate in a survey. They may be also offered to participate in the survey online or by telephone. Furthermore, data from multiple sources are utilized to assist survey data collection, and to supplement and complement data from traditional surveys for inferences. For instance, satellite images are used to prescreen for buildings with certain desired characteristics. Administrative data can be linked to obtain additional information without burdening survey respondents.
This session will be exploring the innovative uses of multimodes and multidata sources to assist survey data collection, reduce burden, improve survey quality, and reduce cost of data collection. This session focuses on challenges, success, and error trade-offs in the multimode and multidata environment. Researchers are invited to submit papers on any of the following topics:
• Challenges, success, and error trade-offs of using multiple modes to contact/reach sampled persons
• Challenges, success, and error trade-offs of using multiple modes to collect data from sampled persons
• Challenges, success, and error trade-offs of using data from different sources to improve sampling efficiency
• Challenges, success, and error trade-offs of using data from different sources to assist and supplement data collection
• Challenges and success of combining survey data across multiple modes and from multiple sources
• Evaluating error trade-offs arising from combining data from multiple modes and/or multiple
Miss Jasmine Mitchell (University of Essex) - Presenting Author
Professor Annette Jäckle (University of Essex)
Surveys are increasingly asking respondents to complete additional data collection tasks that go beyond completing survey questionnaires. These might include tasks embedded within the survey, such as consents for data linkage, and tasks respondents have to complete after the interview, such as a diary or mobile app. In previous research we have started to examine the cumulative effects of such additional tasks on dropout in a panel survey. Our findings suggest that each invitation to an additional task increases the probability of drop out by on average two percentage points. This suggests that asking respondents to complete additional data collection tasks might be detrimental to panel surveys.
In this paper we examine (1) which types of additional tasks increase dropout from annual interviews of a household panel, and (2) which types of respondents are more likely to drop out from the panel if they are invited to additional tasks. We use data from 15 additional tasks across 16 waves of the Understanding Society Innovation Panel. This is a clustered and stratified sample of approximately 1,500 households in Great Britain with refreshment samples added about every three years. The 15 additional tasks include data linkage consent questions, mobile app studies, bio measures and samples, a time-use diary, monthly surveys, and consent to send survey questions by SMS. Our analysis sample includes 6,712 sample members who completed at least one of the annual interviews. Using these data, we will conduct survival analyses to determine which types of tasks increase the probability of subsequent dropout from the panel and which types of people are more likely to drop out due to additional tasks. The findings will contribute to decisions on how best to gather data on different concepts using different methods, in a way that sample members will cooperate.
Dr Paul Beatty (US Census Bureau) - Presenting Author
Ms Emma Farrell (Australian Bureau of Statistics)
Ms Fiona James (UK Office for National Statistics)
The rising costs of interviewer-based surveys require statistical agencies to consider less expensive alternatives, such as adaptation to self-administration and use of auxiliary data. Of course, survey sponsors also seek assurances that such alternatives produce comparable estimates that meet data quality standards. Representatives of several national statistical agencies met in Geelong, Australia in December 2022 to discuss potential approaches to address these problems. This paper highlights subsequent efforts to carry forward a research agenda at three of these agencies: the US Census Bureau, the Australian Bureau of Statistics, and the UK Office for National Statistics. A substantial effort for all three agencies includes evaluating self-administered alternatives to interviewer-based questions. Examples including replacement of open-ended questions that require interviewer probing and follow-up with simpler, self-contained questions; reduction of instructions and navigational complexities; modification of help features from interviewer-based to respondent-based; creation of visual aids for both motivation and clarification; and the use of AI to assist with complex classifications usually performed by interviewers. Follow up studies (involving experiments when possible, but also including qualitative evaluation, paradata and response metrics) evaluate how well alternative versions produce comparable results, and exploring data quality tradeoffs produced by the alternative versions. Also, while some agencies are using web-first administration, others are exploring whether response can be optimized by pushing only subsets of the sample to self-administration. Finally, agencies increasingly explore the quality of alternative data sources to augment or replace survey items, as well as respondent willingness to provide access to these data. The presentation provides examples of each of these initiatives spanning the three agencies, noting both commonalities across research programs as well as distinctive emphases.
Dr Hanyu Sun (Westat) - Presenting Author
Dr Ting Yan (NORC at the University of Chicago )
Dr David Cantor (Westat)
Multimode data collection has become particularly popular in the recent years. The literature has shown that using multiple modes of contact can increase response rates over a single contact mode (Dillman et al. 2014). Most of the previous research has examined the impact of time-invariant factors on survey response such as mode of data collection, contact mode, and respondent characteristics (Coffey, Maslovskaya, McPhee 2024). A few studies have examined how time-varying factors affect response patterns such as date and time of contact (e.g., Fang et al. 2021). There is little research explores the impact of time-varying factors on sample representativeness and the interaction between time-invariant and time-varying factors on survey participation. In this presentation, we will examine whether and how time-varying and time-invariant factors affect survey participation and sample representativeness in four waves of a multimode panel study. Specifically, we will study the effects of time-varying factors (days in the field, day of a week, contact mode) and time-invariant factors (e.g., demographic characteristics) on respondents’ participatory decision, using discrete-time survival analysis to model the effects of time-varying and time-invariant factors on the conditional odds of a person responding to the web survey (given that they have not responded yet). We will also examine the sample representativeness by contact and by wave utilizing sampling frame variables. In addition, we will discuss the implications of the findings and recommendations for future research.
Dr Kristen Olson (University of Nebraska-Lincoln) - Presenting Author
Dr Jolene Smyth (University of Nebraska-Lincoln)
Dr Rebecca Medway (AIR)
Dr Ting Yan (NORC)
Mixed-mode surveys provide tools to survey researchers to balance survey costs and survey errors. When mixing self-administered web and mail survey modes, an open question is whether to offer survey respondents both mail and web surveys concurrently or to order them sequentially. Although initial research on mixed-mode surveys indicated that concurrent mixed-mode designs yield lower response rates than mail-only surveys (Medway and Fulton 2012), more recent work raises questions about whether this finding still holds (Olson, et al. 2021). In this paper, we use meta-analytic methods to examine response rates and mode selection in mail-only, web-only, concurrent mixed-mode, and sequential mixed-mode surveys. We examine 83 published and unpublished studies from 2001 to 2023. Initial findings indicate that web-only response rates are about 14 percentage points lower than mail-only surveys. For early studies (before 2012), we replicate the finding that mail-only surveys have higher response rates than concurrent mixed-mode designs, but the difference in response rates between concurrent mixed-mode designs and mail-only surveys has disappeared in more recent studies. We also find that sequential mixed-mode designs have lower response rates than mail-only studies, on average. We find that the mix and sequence of modes affects the proportion of respondents who participate by web, with about 25% participating by web in a concurrent mixed-mode design – a rate that has increased over time – and over half participating by web in a sequential web-to-mail design. We examine moderating factors at the study and treatment level, including whether the study is longitudinal or cross-sectional, whether email is used as a contact method, and use of incentives, among others. We conclude with implications for survey practice.
Mrs Gina Cheung (Independent) - Presenting Author
Over the past 40 years, Blaise, the advanced data collection tool developed by Statistics Netherlands (CBS), has been widely adopted and successfully utilized by prominent statistical agencies (NSIs), universities, and survey research organizations. Its robust paradata capabilities have significantly influenced survey research methodologies and data collection strategies worldwide, emphasizing the growing importance of paradata in contemporary research.
In response to the pressing challenges of declining response rates faced by many NSIs, Blaise has introduced a Mixed Mode Case Management (MMCM) system and dashboard. This system addresses various challenges by creating a unified platform that integrates all modes of data collection—CAWI, CATI, and CAPI—into a cohesive framework. The initiative is designed to facilitate seamless collaboration among these modes, permitting both sequential and concurrent usage.
A key aspect of this effort is the development of a comprehensive mixed-mode dashboard that provides a clear, unified overview of all data collection activities and outcomes. By streamlining processes and harmonizing terminologies and workflows, we aim to significantly reduce data discrepancies and enhance overall system efficiency. Through this integration, we anticipate improvements in data collection processes and a more intuitive user experience, as well as more reliable and accurate data outputs.
In a later stage, we also plan to integrate CAVI (video interviewing) and other modes, such as paper-based surveys.
In the presentation, we will demonstrate how to effectively use MMCM to manage a mixed-mode project.
Dr Joseph Sakshaug (IAB; LMU-Munich) - Presenting Author
Dr Jan Mackeben (IAB)
Telephone surveys have historically been a popular form of data collection in labor market research and continue to be used to this day. Yet, telephone surveys are con-fronted with many challenges, including imperfect coverage of the target population, low response rates, risk of nonresponse bias, and rising data collection costs. To address these challenges, many telephone surveys have shifted to online and mixed-mode data collection to reduce costs and minimize the risk of coverage and nonresponse biases. However, empirical evaluations of the intended effects of introducing online and mixed-mode data collection in ongoing telephone surveys are lacking. We address this research gap by analyzing a telephone employee survey in Germany, the Linked Personnel Panel (LPP), which experimentally introduced a sequential web-to-telephone mixed-mode design in the refreshment samples of the 4th and 5th waves of the panel. By utilizing ad-ministrative data available for the sampled individuals with and without known tele-phone numbers, we estimate the before-and-after effects of introducing the web mode on coverage and nonresponse rates and biases. We show that the LPP was affected by known telephone number coverage bias for various employee subgroups prior to intro-ducing the web mode, though many of these biases were partially offset by nonresponse bias. Introducing the web-to-telephone design improved the response rate but increased total selection bias, on average, compared to the standard telephone single-mode design. This result was driven by larger nonresponse bias in the web-to-telephone design and partial offsetting of coverage and nonresponse biases in the telephone single-mode de-sign. Significant cost savings (up to 50% per respondent) were evident in the web-to-telephone design.
Mr Mátyás Gerencsér (Hungarian Central Statistical Office) - Presenting Author
It is widely accepted that mixing data collection modes is an effective way to reduce survey nonresponse. This is because different data collection methods may engage different respondent groups, thus improving coverage, reducing the rate of nonresponse errors, and making data collection more cost-effective. However, mode effects may also occur.
Due to declining response rates (RRs), the Hungarian Central Statistical Office designed a field experiment in which a probability sample, selected from the Hungarian registry of addresses, was randomly divided into three groups with different sequential mix-mode data collection protocols. CAWI was the first mode in each group; in Group A it was followed by CATI, then CAPI, in Group B the order of interviewer-administered modes was reversed, whereas in Group C only a single CAPI mode followed. Nonrespondents from all groups were re-invited for a final CAWI phase.
The aim was to reveal how the order of data collection modes may affect the RR, respondents’ characteristics and response quality. Additionally, it was also examined to what extent nonrespondents can be converted by reintroducing a post-fieldwork CAWI option. It was also analyzed how mode effects differ during the second CAWI round from the responses collected in the first CAWI round.
Different sequences led to notable differences in settlement type and income. Other results were generally not significant. In the final phase, when CAWI was reintroduced, 25% of all previous nonrespondents were successfully converted. It contributed greatly to achieving a high overall RR of 72.37%. Group- and mode-specific results of the experiment will also be presented and discussed.
Mrs Lina Lloyd (National Centre for Social Research) - Presenting Author
The English Longitudinal Study of Ageing (ELSA) follows people aged 50 and over. It is a pivotal study helping understand ageing dynamics in the UK for over two decades. With 11 waves of data collected primarily through face-to-face interviews and paper-based self-completions, ELSA is transitioning to a multimode design in Wave 12. This shift is driven by the need for cost-efficiency, declining response rates, and a growing demand for timely, frequent, and high-quality data.
This session explores the use of multimode design in ELSA to improve making contact and following longitudinal survey respondents while addressing the challenges and trade-offs associated with such transition. There is a specific concern with transitioning ELSA to web-first because of the elderly target population for this survey. Because of this the communication strategy incorporates tailored approaches, such as offering flexible methods for updating contact details, direct face-to-face invitations for participants with cognitive impairments, low digital skills or limited internet access. These strategies have been assessed in the planning stages. This includes a dress rehearsal, cognitive and usability testing, focus groups with participants, and literature reviews. Key findings and preliminary results from the 2025 dress rehearsal will be shared.
Particular focus will be given to the challenges of transitioning specific survey components online, such as cognitive assessments and health visits, which are particularly prone to mode effects. To mitigate these, a hybrid approach was adopted: cognitive tasks will be administered via telephone for those completing the main survey online. Health visits will continue to be administered face-to-face every other wave. Keeping some interviewer contact elements in the survey will ensure data consistency while maintaining respondent engagement.
By addressing challenges and sharing lessons learned, this session offers valuable insights into the future of longitudinal surveys.
Ms Sophie Hensgen (Institute for Employment Research) - Presenting Author
Dr Joseph Sakshaug (Institute for Employment Research)
Achieving high response rates is a reoccurring challenge for many surveys. However, voluntary establishment surveys face unique challenges, such as intricate business structures, data security protocols, participation during work hours, and complex questionnaires, which increase the difficulty to collect high quality interviews. Further, these surveys lack the enforcement power of mandatory surveys.
Face-to-face interviews can overcome several challenges associated with data collection in voluntary establishments, offering higher response rates and good data quality. They, however, are expernsive, and the shrinking interviewer workforce further necessitates exploring alternatives such as web-based surveys, or sequential mixed-mode designs. While the latter can help limit costs and increase response rates, it also introduces mode effects. Considering all these factors, identifying the most effective mode design for establishment surveys is essential.
In the 2024 wave of the IAB Establishment Panel an experimental setup was employed in which randomly selected groups were assigned to different single-mode or sequential mixed-mode designs to investigate their effectiveness for the refreshment sample.
The aim of this study is to compare response rates, data quality, and selection bias across the assigned groups to determine the most effective survey design. Additionally, we examine the impact of specific design elements, such as dividing the sample into tranches for better management or setting deadlines, on improving response rates. This study provides practical insights and lessons learned to optimize establishment data collection in an evolving survey landscape.