ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Evaluating Mixed-Mode in Panel Surveys: Where do we stand and where do we go from here 2?
Session Organisers	Dr Patricia Hadler (GESIS - Leibniz Institute for the Social Sciences) Dr Steffen Pötzschke (GESIS - Leibniz Institute for the Social Sciences)
Time	Wednesday 16 July, 11:00 - 12:30
Room	Ruppert 042

Mixed-mode surveys combine different data collection methods such CAPI, CATI, PAPI, and CAWI. Mixed-mode designs (might) offer respondents the opportunity to respond to a survey in their preferred mode, can decrease biases etc. At the same time, mixing modes makes survey design more complex and raises questions regarding the comparability of measurements across modes.

Due to their longitudinal character and the need to safeguard data quality and measurement equivalence over time, mode choice is particularly important to panel studies. While some panel surveys used mixed-mode designs from the start, other, often long-running panel surveys have switched from mainly interviewer-administered data collection to mixed-mode designs employing additional self-administered approaches, or are even considering transitioning to web-only data collection in light of declining response rates and rising costs.

This session invites submissions exploring the use of mixed-mode data collection in panel surveys. Authors are encouraged to present research on the advantages, limitations, and implications of mixed-mode designs. Topics may include, but are not limited to:

- Mode effects: How do different modes impact data quality, measurement error, and response patterns?
- Sampling bias: Do mixed-mode designs improve sample quality?
- Cost-benefit analysis: What are the trade-offs between cost savings and data quality in mixed-mode approaches?
- Attrition: How do mixed-mode designs influence participant retention over time?
- Mode sequencing: What is the optimal mode sequence, and how do transitions affect respondent behaviour?
- Data harmonization: How can researchers ensure data comparability across modes?
- Innovation: How are emerging technologies—like mobile apps and passive data collection—transforming mixed-mode panel surveys?

We welcome empirical studies, theoretical analyses, case studies, and innovative methodologies that evaluate the state of the art of mixed-mode panel surveys and provide insight on possible future developments.

Keywords: mixed-mode surveys, mode effects, survey modes, panel surveys, longitudinal surveys, attrition, data quality

Papers

Unit-Nonresponse in the IAB-Establishment Panel: What can we learn for our fieldwork?

Dr Susanne Kohaut (Institute for Employment Research) - Presenting Author
Dr Iris Möller (Institute for Employment Research)

The IAB-Establishment Panel is the most comprehensive establishment survey in Germany with 15.000 firms participating every year. New firms are also included to cover panel attrition and the change in the economy. Until 2018 all interviews were conducted face-to-face with paper and pencil. In 2018 a computer-aided instrument was introduced (CAWI/CAPI) in an experiment and in the following years sequential modes were offered to the respondents starting with a Web-questionnaire with face-to-face follow-ups.
In recent years response rates dropped considerably. Therefore, we like to gain insight into the unit-nonresponse process and to finde out what role data collection and other factors play. We analyse firms that took part in the survey from 2021 to 2023 which means that we focus on the panel sample of the IAB-Establishment Panel. The advantage of a panel survey is that information is available from previous years. Our conceptual framework is based on the model for the decision process in establishments presented by Willimack/ Snijkers (2013).
Using probit-models we analyse unit-nonresponse in 2023 to figure out which factors drive the probability to take part again in the survey. With the data of the IAB-Establishment Panel a wide range of firm characteristics are available as well as data on the respondent. We can also use paradata like the duration of the interview to model the response process. First result show that a change in the survey mode is associated with a lower response rate in the following year as well as variables that point to a tight economic situation for the firm. On the other hand, firms that are quick to respond and have the same respondent over time are more likely to stay in the survey.

The Effects of Alternative Mixed-Mode Household Screening Protocols and Invitation Letter Envelope Types on Participant Recruitment: Evidence from the Health and Retirement Study

Dr Brady West (University of Michigan) - Presenting Author
Ms Chendi Zhao (University of Michigan)
Mrs Heather Schroeder (University of Michigan)
Mr Paul Burton (University of Michigan)
Mrs Eva Leissou (University of Michigan)
Mr Andrew Hupp (University of Michigan)

With declining response rates in panel surveys worldwide, researchers have been exploring optimal mixed-mode approaches for panel member recruitment. This study considered the household screening stage of recruitment in the 2022 Health and Retirement Study (HRS). The study investigates the effects of different initial contact protocols and invitation letter envelope types on completion outcomes and contact attempts, specifically focusing on the variation in effects among race/ethnicity and age groups. A total of 16,452 sampled households were divided randomly into two groups. One group received a “web-first” protocol, where sampled individuals were invited to complete the screening questionnaire online and nonrespondents were eventually followed up FTF (face-to-face), and the other group received a "field-first" (FTF) protocol. Within the "web-first" group, two random subgroups were created. One subgroup received a regular HRS mailing envelope containing the invitation letter and a pre-paid $2 cash incentive. The other received an envelope with the cash visible.

We analyzed completion outcomes by screening protocol and envelope type. We tested interactions in multivariable models to explore differences in these effects among race and age groups. We also evaluated predictors of the number of contact attempts needed among completed cases. The results indicate that the "web-first" protocol increased completion rates and reduced the contact attempts needed for a completed screener among the youngest individuals. However, this protocol had an opposite effect among Hispanic individuals. The envelope type also significantly influenced screener completion. The regular envelope increased completion rates among younger and non-Hispanic Black individuals, and the youngest age group also generally received fewer contact attempts before completion. Overall, the findings highlight the effectiveness of employing different contact methods using different modes to enhance panel survey participation, specifically among different socio-demographic subgroups.

Influence of Survey Administration Mode on Well-Being, Health, and Illness Outcomes: A Randomized Study

Miss Marie Stjerne Grønkjær (Center for Clinical Research and Prevention, Copenhagen University Hospital – Bispebjerg and Frederiksberg, Copenhagen) - Presenting Author
Mrs Marie Holm Eliasen (Center for Clinical Research and Prevention, Copenhagen University Hospital – Bispebjerg and Frederiksberg, Copenhagen)
Mr Peter Elsborg (Center for Clinical Research and Prevention, Copenhagen University Hospital – Bispebjerg and Frederiksberg, Copenhagen)
Mrs Anne Helms Andreasen (Center for Clinical Research and Prevention, Copenhagen University Hospital – Bispebjerg and Frederiksberg, Copenhagen)

Background: Participation is crucial in large-scale population surveys that aim to draw valid conclusions about the general population. In a previous study, we found that participation rates were substantially influenced by the survey administration mode and the number of reminders, while sociodemographic representativeness was largely unaffected. Using the same study sample and a randomized design, this study aimed to examine whether the survey administration mode influenced key survey outcomes related to well-being, health, and illness.

Methods: We used data from the panel study of The Danish Capital Region Health Survey, including 6564 individuals (≥20 years) who participated in both 2017 and 2021. A sequential mixed-mode survey administration was used in 2017, whereas individuals were randomized into two groups in 2021: a single-mode group (three digital letters) and a sequential mixed-mode group (two digital and then three physical letters). Self-reported survey data on well-being, health, health behavior, and illness in 2021 were compared between the two groups.

Results: When comparing key outcomes between the single-mode group (N=2876) and the sequential mixed-mode group (N=3688), poor quality of life was more prevalent in the single-mode group (4.4%) than in the mixed-mode group (3.3%; p=0.016). The groups showed no significant differences in loneliness, stress symptoms, health-related outcomes (self-rated health, obesity, sleep problems, smoking, alcohol consumption, diet, physical activity), or chronic disease prevalence.

Conclusion: Differences between the survey administration groups were observed in only one of several key outcomes. Thus, in this panel study sample, key outcomes related to well-being, health, and illness were largely unaffected by survey administration mode.

Introducing Mixed-Mode in a Refugee Household Panel – An Initial Evaluation

Ms Theresa Müller (infas Institut für angewandte Sozialwissenschaft GmbH (Institute for Applied Social Sciences)) - Presenting Author
Mr Michael Ruland (infas Institut für angewandte Sozialwissenschaft GmbH (Institute for Applied Social Sciences))
Dr Elena Sommer (Socio-Economic Panel at DIW)
Professor Sabine Zinn (DIW-SOEP / Humboldt University Berlin)

The IAB-BAMF-SOEP Survey of Refugees in Germany is an annual household panel conducted as part of the Socio-Economic Panel (SOEP) since 2016. Its primary data collection method has always been CAPI, at both the household and individual level. Households are approached by interviewers, and until recently, at least the household interview was required to be conducted face-to-face but there are options to switch to a self-administered mode (CAWI or CASI) at the individual level.

In the 2024 survey wave, households were, for the first time, invited to complete the entire survey online as part of a CAWI follow-up. This sequential mixed-mode design at the household level was introduced during the final phase of fieldwork to boost response among households, that were hard to reach or motivate.

One the one hand, this paper evaluates which households could be reached through the CAWI follow-up and whether panel attrition could be reduced by offering this additional mode. On the other hand, we compare data quality indicators between households surveyed via CAPI and CAWI. Furthermore, we analyze which households should be offered the option to switch to CAWI at the household level and at which point in the field period this switch offer could be most effective.

Making the switch. Evidencing the differences in response rates, data quality and costs between face-to-face-only and sequential mixed-mode (web>F2F) in the UK Millennium Cohort Study Age 23 Survey

Dr Larissa Pople (UCL Centre for Longitudinal Studies) - Presenting Author
Mr Matt Brown (UCL Centre for Longitudinal Studies)
Professor Emla Fitzsimons (UCL Centre for Longitudinal Studies)
Mrs Lucy Haselden-Garcez (UCL Centre for Longitudinal Studies)
Mr Nicholas Gilby (Ipsos)
Mrs Kirsty Burston (Ipsos)

In recent years longitudinal studies have made increasing use of mixed-mode data collection strategies, especially those involving the web. This shift has been driven by falling response rates, increasing survey costs for face-to-face surveys, and almost universal internet access in the population.

The UK Millennium Cohort Study (MCS) is a multi-disciplinary longitudinal study following the lives of around 19,000 children born in 2000-2002 across the UK.

The 8th sweep, the Age 23 Survey, involves a multifaceted questionnaire that includes cognitive assessments, the collection of consents for administrative data linkage and other elements. All previous sweeps were conducted face-to-face, enabling the collection of anthropometric and other complex measurements.

Finding and engaging participants at this critical juncture - the first major adult wave in which participants will be approached fully independently from parents - is vital to the ongoing success of the study. Participants are expected to be highly mobile and living busy lives. Perhaps now is the right time to switch to a web-first, mixed-mode approach?

To inform the decision we conducted an experiment in the first phase of fieldwork, randomly allocating half of the sample to take part face-to-face and half to take part via web, with non-respondents followed up face-to-face.

In this paper we explore the findings of this experiment. We compare the overall response rates achieved in the two arms of the experiment. We also explore differences in sample composition, data quality, item-non-response, interview length, mode effects on measurement, levels of interviewer effort and fieldwork costs across the two approaches.

Who is still offline? An analysis of web non-users and web non-respondents with a longitudinal perspective

Dr Pablo Cabrera-Álvarez (University of Essex) - Presenting Author
Dr Jonathan Burton (University of Essex)
Professor Annette Jäckle (University of Essex)
Professor Gabriele Durrant (University of Southampton)
Dr Jamie Moore (University of Essex)
Professor Peter Smith (University of Southampton)

The last decade has seen some high-quality surveys adopting web as the primary mode for data collection, a trend that has been accelerated due to the pandemic. To reach segments of the population without internet access, most high-quality surveys employ mixed-mode designs, pairing web surveys with an interviewer-administered mode such as face-to-face or telephone. However, the mixed-mode designs entail higher fixed costs than a web-only survey and might introduce measurement differences affecting data comparability. As internet access has grown in the UK to cover almost the whole population and the levels of digital literacy have increased considerably, a critical question is whether it is now feasible to conduct web-only surveys of the general population without compromising representativeness.
This research first examines the characteristics of the UK population that remains offline and how they differ from those who use the internet. Second, it evaluates differences between web survey respondents and non-respondents, exploring how these differences affect representativeness. This analysis aims to disentangle the bias due to the offline population from the bias produced by the rest of the web non-respondents. The analysis uses data from the Innovation Panel and the main study of Understanding Society, the United Kingdom Household Longitudinal Study (UKHLS). We use the Innovation Panel, which includes a random subsample of households allocated to a sequential web-first with CAPI follow-up mode design since 2012, to explore the research questions with a longitudinal perspective. Furthermore, we use a larger random subsample of the UKHLS main study that transitioned to a web-first design in 2020 due to the COVID-19 pandemic to investigate the effect of using a web-only design on the of some subgroups, such as ethnic minorities.