ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Adapting survey mode in a changing survey landscape: Experiences from repeat cross-national, cross-sectional, and general social surveys 4
Session Organisers	Dr Gijs van Houten (Eurofound) Dr René Bautista (NORC at the University of Chicago) Professor Rory Fitzgerald (ESS HQ; City St Georges, University of London) Mr Tim Hanson (ESS HQ; City St Georges, University of London) Mr Nathan Reece (ESS HQ; City St Georges, University of London) Ms Daphne Ahrendt (Eurofound) Ms Jodie Smylie (NORC at the University of Chicago)
Time	Wednesday 16 July, 11:00 - 12:15
Room	Ruppert Blauw - 0.53

Studies to measure attitudes, opinions, and behaviors are critical to understanding societies around the world. In the face of social developments, changing trends in respondent recruitment methods, budget constraints, national infrastructure disruptions, and public health concerns, many repeat cross-sectional social surveys are experimenting with self-completion and mixed-mode approaches. The European Social Survey (launched 2001), United States’ General Social Survey (launched 1972), and the European Quality of Life Surveys (launched 2003) are examples of longstanding studies collecting data to inform research on changes over time and now exploring and transitioning to new modes. This session brings together cross-sectional social surveys to share experiences in survey mode transition.

The session's aims include: (1) Share results and lessons from recent mode experiments and mixed-mode applications by general social studies, and potential ways to improve methods. (2) Highlight how different cross-sectional studies have recently modified survey protocols to adapt to changing public conditions. (3) Provide space for data creators, data users, and survey practitioners to discuss methodological and statistical challenges for cross-sectional studies considering such moves. (4) Discuss integrity and comparability of data collected using new data collection methods with the existing time-series. (5) Explore applications of emergent technologies to new modes.

We invite submissions from those involved in transitioning repeat, cross-sectional, and cross-national social surveys to new data collection approaches. Topics of interest include: results from pilots or feasibility studies based on self-completion or mixed-mode approaches; findings from experimental research testing aspects of self-completion/mixed-mode designs (e.g., incentive and mailing strategies, survey length adaptations, sequential vs. concurrent designs); impacts of mode switches on measurement and survey time series; and discussions of experiences and challenges with adapting cross-sectional surveys to new modes across different cultural/national contexts.

Keywords: general surveys, survey methodology, data collection, data collection modes, mixed mode, self administration

Papers

Moving an employment survey from a face-to-face design with a fresh sample to an online panel: Evidence from a parallel run in the UK

Mr Curtis Jessop (The National Centre for Social Research) - Presenting Author
Ms Sarah Butt (The National Centre for Social Research)
Mr Simon Moss (The National Centre for Social Research)
Ms Jo D'Ardenne (The National Centre for Social Research)

Since 1986 the Skills and Employment Survey has collected robust survey data on the skills and employment experiences of people aged 20-65 working in Britain. In response to changes in societal expectations about how they provide data, and to future-proof the study, in 2023 the decision was made to explore the possibility of transitioning to an online design. Data were collected in parallel both via a face-to-face interview with a fresh sample taken from the postcode address file (PAF) and using a sequential mixed-mode (web/telephone) design with sample from a probability-based panel.

This paper will summarise the two designs, the approach taken to transitioning the questionnaire, and outline findings from the parallel run. It will look firstly at how the different sampling and fieldwork designs impacted response rates and the participating sample profile. It will then look at the extent to which the two arms of the study produced different estimates across the survey. It will also examine whether those differences can be explained by differences in the sample profile and/or whether or not there are any patterns in the differences – are certain types of questions more likely to produce different estimates than others, and how does that align with current literature on the impact of survey mode on measurement?

These findings will provide evidence on how transitioning from a face-to-face survey conducted with a fresh sample to an online panel survey may impact estimates, and what types of questions may be more or less at risk of mode effects.

Do web surveys of the general population still require follow-ups of non-respondents in other modes to ensure representativeness? A case study using Understanding Society – the UK Household Longitudinal Study.

Dr Jamie Moore (ISER, University of Essex) - Presenting Author
Dr Pablo Cabrera-Álvarez (Institute for Social and Economic Research)
Professor Gabriele Durrant (University of Southampton)
Professor Annette Jackle (ISER, University of Essex)
Dr Jon Burton (ISER, University of Essex)
Professor Peter Smith (University of Southampton)

Many social surveys are facing significant challenges, such as increasing nonresponse and survey costs. One approach to minimising costs is to shift from face to face (F2F) or telephone interviews to less costly web interviews. Research has shown that while there may be benefits to web mode in terms of responses from individuals not sampled by F2F or telephone modes, generally non-response biases are minimised by offering both. Consequently, some surveys have adopted designs in which sample members are first offered web mode, then non-respondents are followed up with F2F or telephone. However, in recent years proportions of populations using the internet have increased markedly and people have become less willing to welcome interviewers into their homes. Hence, it is possible that the benefits of F2F or telephone follow ups in terms of reducing non-response biases, the justification for such designs, have changed. We investigate this question in datasets from Understanding Society - the UK Household Longitudinal Study (UKHLS). We focus on the Innovation Panel component of the study, in which a subset of sample members has been offered web interviews with F2F or telephone follow-ups of non-respondents since 2012. We evaluate 1) the impact of following up web non-respondents on how well respondent datasets reflect sample datasets (dataset representativeness), and 2) how the impact of these follow-ups has changed over time. For each survey wave, we use Coefficients of Variation of response propensities to quantify the representativeness of web only and web plus F2F or telephone respondents compared to the sample in terms of survey measured characteristics. In addition, we repeat our analyses using the UKHLS main survey dataset, whose more representative composition enables consideration of whether patterns differ for hard-to-reach population elements.

PUSHING OLDER TARGET PERSONS TO THE WEB: RESULTS OF A MODE CHOICE SEQUENCE EXPERIMENT IN THE EUROPEAN SOCIAL SURVEY IN GERMANY

Dr Jan-Lucas Schanze (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Mrs Caroline Hahn (GESIS - Leibniz Institute for the Social Sciences)
Dr Oshrat Hochman (GESIS - Leibniz Institute for the Social Sciences)

While a sequential, push-to-web mode sequence is very well established in survey research and commonly used in survey practice, many social surveys still prefer to contact older target persons with a concurrent design, offering a paper questionnaire alongside a web-based questionnaire from the first letter onwards. In our presentation, we compare the performance of a sequential design with a concurrent design for target persons older than 60 years. We analyse response rates and compare the sample composition and distributions in substantive items within resulting net samples.

Data stems from the 10th Round of the European Social Survey (ESS) carried out in self-completion modes (CAWI/PAPI) in some countries in 2021. In Germany, a mode choice sequence experiment was implemented for all target persons older than 60 years. 50% of this group was invited with a push-to-web approach, offering a paper questionnaire in the third mailing. The control group was invited with a concurrent mode sequence, offering both modes from the beginning on.

Results shows similar response rates for the concurrent design and the sequential design (AAPOR RR2: 38.4% vs. 37.3%). This difference is not statistically significant. In the concurrent group, 21% of the respondents answered the questionnaire online, while in the sequential group this was the case for 50% of all respondents. Even among target persons older than 75 years, every third respondent took part in the web questionnaire when being pushed to the web. The resulting net samples are very comparable. Looking at various demographic, socio-economic, attitudinal, and behavioural items, no significant differences were found between the sequential and concurrent groups. We conclude that a significant share of the elderly German population can be pushed to the web without negative consequences for response rate or sample composition.

Transitioning from face-to-face to self-completion mode: insights from ISSP and ESS surveys in Spain (2014-2024)

Dr Mónica Méndez-Lago (CIS-Centro de Investigaciones Sociológicas) - Presenting Author

This presentation explores the transformation in survey administration modes for the International Social Survey Program (ISSP) and the European Social Survey (ESS) in Spain from 2014 to the present. In 2014, the ISSP Citizenship module was administered both face-to-face and a mixed self-completion mode, with a "Push-to-Web + mail" approach. As far as we know, this was the first time a social/political attitude survey in Spain used this latter design. At that time, Internet penetration in Spain stood at approximately 76%, with significantly lower figures among the population 50 years old and over. Since then, Internet penetration has risen to 95% of the overall population, with significant reductions in disparities related to Internet usage and familiarity. However, a notable gap remains among people aged 65 and older, who continue to have lower digital coverage.

Since 2020 ISSP surveys in Spain are conducted exclusively using a self-completion mix mode (Push-to-web+mail). This presentation will analyze key aspects of this shift over the last decade, including the evolution of overall response rates and specific trends within different population groups, with a particular focus on individuals over 65 and/or low education attainment, comparing these figures with other modes of administration. It will also present comparative evidence with the ESS, which has traditionally been conducted using a face-to-face mode, except for round 10 (2022) when Spain adopted a Push-to-web+mail design due to the COVID-19 pandemic.

The presentation will also deal with a central question in the current design of self-completion surveys, namely whether offering paper-based response options remains necessary, and for what purpose. This case study of Spain in the last decade offers interesting insights into the broader challenges and opportunities facing survey research.