ESRA 2025 Preliminary Program

All time references are in CEST

Sampling mode comparisons
Session Organiser	Dr Daniel Seddig (KFN)
Time	Thursday 17 July, 15:30 - 17:00
Room	Ruppert 011

Papers

Comparison of probabilistic and non-probabilistic samples in the European Training and Learning Survey

Ms Alexandra Cronberg (Verian) - Presenting Author
Mr Jamie Burnett (Verian)

This study, conducted by Verian on behalf of the European Centre for the Development of Vocational Training, sought to gain insight into adult learning and development across the 27 EU member states, plus Iceland and Norway. The study combined probabilistic samples (online panel or telephone RDD) and non-probabilistic samples (online access panels) in each country to enhance sampling efficiency.

This paper seeks to answer the following questions:
1. To what extent were systematic differences observed in key outcome indicators between the probabilistic and non-probabilistic samples in the 29 study countries, based on data weighted for demographics?
2. To what extent were differences in key outcome indicators between the probabilistic and non-probabilistic panels reduced by adding a non-probability weight, adjusting the non-probability sample to become more similar to the probability sample?

The study employed non-probabilistic online panels in all countries, and telephone RDD in 11 countries and probabilistic online panels in 18 countries (n=44,752 overall). The samples were equally split between the probabilistic and non-probabilistic approach in each country.

Differences between the probabilistic and non-probabilistic samples within each country were analysed using cross-tabulations, Chi-square test, and logistic regression analysis using data weighted for gender, age, education, occupation, industry, and region.
An additional weight was created for the non-probability sample using variables that were significantly associated with the outcome variables, other theoretically relevant variables, and sample type, and in the expected direction. Weights were derived using RIM weighting.

For most countries, differences were observed in key outcomes between the two sample types, although the magnitude and direction varied across indicators as well as countries.
The difference between the probabilistic and non-probabilistic samples were generally reduced by the non-probability weighting approach, though not fully removed.

How to survey a city? The sampling and mode repertoire in the Vienna Quality of Life Study

Dr Julian Aichholzer (IFES - Institut für empirische Sozialforschung) - Presenting Author
Ms Teresa Schaup (IFES - Institut für empirische Sozialforschung)
Dr Reinhard Raml (IFES - Institut für empirische Sozialforschung)
Dr Eva Zeglovits (IFES - Institut für empirische Sozialforschung)

The Vienna Quality of Life Study (WLQS) has been conducted since 1995. It captures various aspects of the quality of life of the Viennese population, providing a comprehensive basis for evidence-based political and administrative decisions.

In this presentation, we describe the methodological approach of the 2023 study, which included more than 8,500 respondents. This year’s survey expanded the methodological repertoire to attract respondents through various channels or methods: in addition to classic telephone interviews (CATI), online interviews (CAWI) from access panels and via postal invitations (Push-to-Web) were used, as well as paper questionnaires and personal (face-to-face, CAPI) interviews. Interviews in foreign languages—Turkish, Croatian, and English—were also possible.

Central aims were to ensure a stratified random sample (minimum sampling size for each city district), balance the accessibility of certain social groups across survey modes, and attract enough respondents from municipal housing complexes (social housing).

We detail our tailored sampling approach, provide examples of the sample structure across different modes, and show method effects on key variables that had to be taken into account.

Probability-Based vs. Non-Probability Online Panel Surveys: Assessing Accuracy, Response Quality, and Survey Professionalism

Dr Sylvia Kritzinger (University of Vienna)
Dr Katharina Pfaff (University of Vienna) - Presenting Author
Mr Max Gschwandtner (University of Vienna)
Dr Julia Partheymüller (University of Vienna)

While probability-based sampling is generally preferred in academic research, the use of data from non-probability online panels has become increasingly common. However, the implications of relying on such samples remain insufficiently understood. In this study, we compare data from two online surveys—the probability-based Digitize! Online Panel Survey and the non-probability AUTNES Online Panel Survey—conducted in Austria following the 2024 parliamentary election. The analysis focuses on evaluating the accuracy of each sample with respect to external benchmarks, such as sociodemographic characteristics and election outcomes. Additionally, we assess response quality indicators, including satisficing, inattentiveness, attitude consistency, acquiescence bias, and non-response rates. Survey professionalism metrics—such as response time, frequency of survey participation, types of devices used, and digital literacy—are also examined. Our findings will clarify the extent to which probability-based samples offer advantages in terms of accuracy, response quality, and survey professionalism when compared to non-probability samples.

How Much Time is Enough?: Evaluating the Trade-off Between Time in Field and Benchmark Accuracy

Ms Megan A. Hendrich (Ipsos Public Affairs) - Presenting Author
Professor Randall K. Thomas (Ipsos Public Affairs)

People who respond earlier to a survey invitation can differ from those who respond later (e.g., Fricker & Tourangeau, 2010). For example, we often find that older, female, urban, and White respondents are more likely to respond earlier in a field period. Though younger, male, rural, Black, and Hispanic respondents respond in lower proportions early on, these harder-to-reach groups tend to increase participation with longer time in field. In collaboration with the New York Times, we conducted a study on the 2022 U.S. midterm election in Wisconsin to investigate how early responders differ from the overall sample in terms of sample composition and bias (compared to high-quality benchmarks, such as Census and state registration data). We fielded two samples in parallel—the first sample included 1,086 web-based completes from an online probability-based panel, and the second sample included 1,610 completes from an address-based sample primarily using a mail-back survey. For each sample type, we compared the total sample with two different subsamples: the first 50% of each sample and the first 75% of each sample. We examined differences in the composition of the early responder samples to the overall sample by comparing demographics and responses to behavioral questions, including vote choice. Average bias somewhat declined as the proportion of the sample completing the survey increased. Bias was reduced both for variables needed for weighting and for other benchmark variables, reflecting improved accuracy with increased time in field. However, the reductions in bias were relatively small. We discuss how a longer field period might improve the accuracy of results, but the value of small increases in accuracy may be outweighed by the value of a shorter period, depending on the study’s goals.

Differences between a CATI and a CAWI survey using the same questionnaire on willingness to share health data and receive vaccinations

Professor Rainer Schnell (University of Duisburg-Essen) - Presenting Author
Dr Jonas Klingwort (Statistics Netherlands)
Professor Sonja Haug (OTH Regensburg)

Reluctance to share medical data and vaccine scepticism in some subgroups of the general population poses challenges to public health strategies. Therefore, the results of surveys are important for political discussions and decisions. Since the media reports often neglect to report technical details of surveys, the results of surveys based on different modes or sampling frames are widely regarded as exchangeable.

To test the implicit assumption of no mode or sampling effects between CATI and CAWI surveys on health policy, we fielded two independent surveys on the willingness to share health data and receive vaccinations. The surveys (n=1308, n=2017) were commissioned in mid-2022 and used the same questionnaire.

We find significant differences with strong effect sizes in marginals and strong differences in the statistical models explaining the dependent variables. Furthermore, large discrepancies in descriptive statistics regarding willingness to share and vaccine scepticism are observed.

These results have practical implications for policymakers and journalists. The publication of survey results on population attitudes toward public health strategies should report the technical details of the surveys in all cases. It should be reported clearly which results depend on CATI surveys and which are due to CAWI surveys, especially when these rely on access panels.

Mode effects in the transition towards self-completion in the European Social Survey

Dr Peter Lugtig (Utrecht University) - Presenting Author

The isolation of the causal effect of survey mode on measurement is difficult due to the fact that selection and measurement effects are (potentially) correlated. Some studies have tried to eliminate selection effects by for example re-interviewing face-to-face respondents in a self-interviewing mode shortly after the original interview. Or they have randomized respondents into a survey mode only after successfully recruiting a respondent into the survey.
In this presentation I will use data from the European Social Survey rounds 9 and 10 to investigate mode effects. In round 9, all countries used face-to-face interviewing. In round 10, nine countries used a self-interviewing instrument (web and paper) with the twentytwo other countries using face-to-face interviewing. The change in survey modes was mostly due to the effects of Covid-19 had on the ability to conduct in-person interviews in some countries. The quasi-experimental design however does allow us to compare countries that switched to self-interviewing with countries that kept using face-to-face interviewing.
Results show that we find no large effects of changing interviewing modes on means, variances and covariances across 111 variables that were measured in both rounds 9 and 10 of the ESS. There are approximately 25 variables where we find effect sizes in the change in means associated with the mode switch is .20 (hedges g) or larger, indicating that there are some variables for which we find mode effects. Additional analyses on experimental data from Great Britain and Finland give further evidence that these effects are probably due to mode measurement. We will in the conference presentation show for what particular variables we find mode effects, and also zoom in differences across countries