ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Adapting survey mode in a changing survey landscape: Experiences from repeat cross-national, cross-sectional, and general social surveys 2
Session Organisers	Dr Gijs van Houten (Eurofound) Dr René Bautista (NORC at the University of Chicago) Professor Rory Fitzgerald (ESS HQ; City St Georges, University of London) Mr Tim Hanson (ESS HQ; City St Georges, University of London) Mr Nathan Reece (ESS HQ; City St Georges, University of London) Ms Daphne Ahrendt (Eurofound) Ms Jodie Smylie (NORC at the University of Chicago)
Time	Tuesday 15 July, 15:30 - 17:00
Room	Ruppert Blauw - 0.53

Studies to measure attitudes, opinions, and behaviors are critical to understanding societies around the world. In the face of social developments, changing trends in respondent recruitment methods, budget constraints, national infrastructure disruptions, and public health concerns, many repeat cross-sectional social surveys are experimenting with self-completion and mixed-mode approaches. The European Social Survey (launched 2001), United States’ General Social Survey (launched 1972), and the European Quality of Life Surveys (launched 2003) are examples of longstanding studies collecting data to inform research on changes over time and now exploring and transitioning to new modes. This session brings together cross-sectional social surveys to share experiences in survey mode transition.

The session's aims include: (1) Share results and lessons from recent mode experiments and mixed-mode applications by general social studies, and potential ways to improve methods. (2) Highlight how different cross-sectional studies have recently modified survey protocols to adapt to changing public conditions. (3) Provide space for data creators, data users, and survey practitioners to discuss methodological and statistical challenges for cross-sectional studies considering such moves. (4) Discuss integrity and comparability of data collected using new data collection methods with the existing time-series. (5) Explore applications of emergent technologies to new modes.

We invite submissions from those involved in transitioning repeat, cross-sectional, and cross-national social surveys to new data collection approaches. Topics of interest include: results from pilots or feasibility studies based on self-completion or mixed-mode approaches; findings from experimental research testing aspects of self-completion/mixed-mode designs (e.g., incentive and mailing strategies, survey length adaptations, sequential vs. concurrent designs); impacts of mode switches on measurement and survey time series; and discussions of experiences and challenges with adapting cross-sectional surveys to new modes across different cultural/national contexts.

Keywords: general surveys, survey methodology, data collection, data collection modes, mixed mode, self administration

Papers

Transitioning the Icelandic National Election Study from Telephone to Mixed-Mode Data Collection

Miss Sara Finnbogadottir (Aarhus University) - Presenting Author
Dr Hafsteinn Einarsson (University of Iceland)

Repeated cross-sectional surveys, such as election studies, benefit from maintaining methodological continuity over time but are facing increasing challenges in the form of declining response rates and increasing costs. This has spurred many longstanding surveys to transition from single-mode to mixed-mode data collection. Here, we explore the effects of transitioning the Icelandic National Election Study, which has been fielded after every national election since 1983, from telephone to mixed mode data collection. In 2021, web response was offered for two sample sub-groups: those who were not associated with known telephone numbers and those who refused to participate over telephone. This survey design maintained telephone as the primary response mode while recruiting low-propensity respondents with other modes. We explore survey out comes in terms of response rates, sample composition and measurement quality and equivalence between telephone and web responses. Our findings suggest that the changes to the survey design were not sufficient to halt the decline in response rates over time but nevertheless resulted in additional responses gathered than a telephone-only design. Web responses were associated with higher rates of non-substantive responses relative to telephone response indicating lower data quality. Overall, these findings suggest that transitioning to a mixed-mode data collection protocol provided some benefits to ICENES but is not on its own sufficient to mitigate the trend of declining response rates over time.

Surveys at the “cross-modes”: Leveraging ESS Round 10 natural experiment to compare response behaviours between face-to-face and self-completion modes

Dr Marek Muszyński (Institute of Philosophy and Sociology, Polish Academy of Sciences) - Presenting Author
Professor Piotr Jabkowski (Faculty of Sociology, Adam Mickiewicz University, Poznan)

Rising costs and challenges in interviewer-administered surveys necessitate the development of mixed-mode studies. Cross-mode comparisons are essential to maintain consistent measurement quality across modes. Research indicates that mode differences affect response behaviours, including attentiveness (Daikeler et al., 2024; Kim et al., 2019) and response styles (Aichholzer, 2013; Hope et al., 2022; Liu, 2015).

The COVID-19 pandemic disrupted the European Social Survey (ESS) Round 10, planned for late 2020 and early 2021, leading some countries to adopt self-completion methods instead of face-to-face interviews. This shift created a natural experiment, enabling the comparison of response behaviours across different survey modes. Our study capitalises on this opportunity to analyse response patterns such as inattentive responding and response styles in ESS Rounds 9 and 10, contrasting the primarily face-to-face mode in Round 9 with the mixed modes of Round 10, where about a quarter of the countries used self-completion modes (postal and web surveys). This is further compared to Round 11, which returned to face-to-face mode.

We employed multilevel regression models to examine cross-mode response behaviours. Outcome variables included straightlining indices and measures of response styles (extreme, midpoint, and acquiescent), with survey mode and socio-demographics (age, gender, education, etc.) as moderators.

Preliminary findings indicate that respondents in self-completion modes exhibit less straightlining and fewer acquiescent and midpoint responses than in face-to-face modes, but yield more extreme responses and outliers identified by Mahalanobis distance. These differences are more pronounced with 11- compared to 5-point rating scales. Mode effects were held when controlled for participant socio-demographic characteristics. Interaction effects between mode and socio-demographics were mainly non-significant.

The findings are significant for designing mixed-mode international surveys and theoretical understanding of mode differences in response behaviours.

Exploring the impact of Mixed-Mode Effects in the Generation and Gender Survey (GGS)

Miss Ilaria Francesca Lunardelli (NIDI - KNAW) - Presenting Author

The second round of the Generations and Gender Survey (GGS) introduced Computer-Assisted Web Interviewing (CAWI) as the primary data collection mode, marking a significant shift towards digital methods in sensitive survey research.
While most countries implemented CAWI as the sole mode for GGS-II Wave 1, a subset opted for mixed-mode data collection, incorporating CAWI alongside other modes such as Computer-Assisted Telephone Interviewing (CATI), Paper and Pencil Interviewing (PAPI), or Computer-Assisted Personal Interviewing (CAPI).
The choice to use multiple modes may enhance sample diversity and inclusion, yet it also introduces potential mixed-mode effects. These effects are particularly critical when examining sensitive indicators, such as fertility and well-being metrics, in a longitudinal, cross-country framework.
To examine the impact of mixed-mode effects, this study will employ a combination of descriptive analyses, regression models and statistical tests within a cross-country comparative framework. The analysis will investigate how mixed-mode effects manifest across different geographical and cultural contexts, such as the French and Uruguayan one, to understand whether these settings influence the extent and nature of mode-related biases. Ultimately, special attention will be given to sensitive indicators, as they are particularly susceptible to mixed-mode effects and hold central importance in in GGS.
Preliminary results suggest that, overall, mixed-mode data collection has contributed to higher response rates and greater diversity among respondents. However, mode effects can still be detected, with variations depending on the country and greater impact on sensitive indicators.
The GGP Central Hub Team aims to provide insights into the implications of mixed-mode effects on accuracy and comparability, as well as contributing to the optimization of future data collection.

The Impact of Switching to Self-Completion Protocols on Item Nonresponse in Complex and Sensitive Survey Questions. Lesson learned from the ESS rounds 9, 10, and 11

Dr Piotr Jabkowski (Adam Mickiewicz University, Poznan) - Presenting Author
Dr Piotr Cichocki (Adam Mickiewicz University, Poznan)
Dr Aneta Piekut (The University of Sheffield)

The COVID-19 pandemic forced a shift in data collection methods, with many surveys adopting self-completion modes to overcome social distance constraints. This study examines the consequences of the transition toward self-completion protocols in the European Social Survey (ESS) and aims to answer whether the transition impacts survey quality. Notably, we focus on item nonresponse rates in complex and sensitive questions. Our analysis works on data from 23 countries participating in the ESS round 9 (2018, face-to-face modes), round 10 (2020, self-completion and face-to-face protocols), and round 11 (2022, face-to-face modes).

We focus on item nonresponse in two types of questions: complex items, such as household composition (gender, age and the relationship with respondent), and sensitive items, such as income and several measures of political trust. Using multilevel regression models, we predicted the probability of item nonresponse occurrence across self-completion (web, paper) and face-to-face modes of data collection (PAPI, CAPI, Video: web), controlling for individual-level predictors (e.g. age, education, gender) and several country-level characteristics.

Our results show differences in nonresponse patterns between face-to-face and self-completion modes. Compared to rounds 9 and 11, when countries utilised self-completion protocols in Round 10, they exhibited lower nonresponse for sensitive items but higher nonresponse rates in complex questions. In contrast, when countries constantly used face-to-face modes for data collection across three rounds, they did not demonstrate any significant differences in item nonresponse rates over time.

Our findings highlight the need for tailored methodological adaptations when moving to self-completion protocols, particularly addressing potential biases introduced by item nonresponse. Our analysis contributes to survey methodology by providing insights for optimising mixed-mode designs, ensuring data quality, and maintaining data comparability over time.

Evaluating the existence and impact of measurement effects in a mixed-mode survey: the case of the French Health Barometer

Mrs Axelle Quiviger (Santé publique France) - Presenting Author
Mrs Noémie Soullier (Santé publique France)
Mr Jean-Baptiste Richard (Santé publique France)
Mrs Leïla Saboni (Santé publique France)
Mrs Maria El Haddad (Santé publique France)

The French Health Barometer is a cross-sectional repeated survey, interviewing the population living in France about their opinions, behaviours and knowledge related to health. In 2024, the survey changed its methodology, transitioning from an exclusively phone interview collection to a mixed-mode data collection combining internet and telephone. To assess the existence and extent of potential measurement effects, a pilot survey was conducted in 2023 in mainland France among individuals aged 18 to 85. This survey tested several protocols simultaneously, including one using only telephone interviews and another using only internet interviews. This random assignment to a mode of data collection allows us to operate within the experimental framework described by Rosembaum and Rubin (1983). Approximately thirty indicators were studied, grouped into four categories depending on which measurement effect was expected: sensitive questions, non-sensitive subjective questions, difficult questions, and factual questions. Three methods were implemented for each indicator: multivariate logistic regressions with the data collection mode as an explanatory variable, propensity score matching, and multivariate logistic regressions weighted by the inverse of the propensity score. To account as much as possible for selection on observables, a large number of explanatory variables were included in the models. Our results demonstrate measurement effects consistent with the literature (satisficing bias, social desirability bias, etc.), but we also found unexpected effects, for which we suggest plausible explanations. The aim of this paper is also to detangle measurement effects from selection effects and to provide questionnaire design recommendations to minimize measurement effects.