ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Assessing the Quality of Survey Data 3
Session Organiser	Professor Jörg Blasius (University of Bonn)
Time	Wednesday 16 July, 09:00 - 10:30
Room	Ruppert paars - 0.44

This session will provide a series of original investigations on data quality in both national and international contexts. The starting premise is that all survey data contain a mixture of substantive and methodologically-induced variation. Most current work focuses primarily on random measurement error, which is usually treated as normally distributed. However, there are a large number of different kinds of systematic measurement errors, or more precisely, there are many different sources of methodologically-induced variation and all of them may have a strong influence on the “substantive” solutions. To the sources of methodologically-induced variation belong response sets and response styles, misunderstandings of questions, translation and coding errors, uneven standards between the research institutes involved in the data collection (especially in cross-national research), item- and unit non-response, as well as faked interviews. We will consider data as of high quality in case the methodologically-induced variation is low, i.e. the differences in responses can be interpreted based on theoretical assumptions in the given area of research. The aim of the session is to discuss different sources of methodologically-induced variation in survey research, how to detect them and the effects they have on the substantive findings.

Keywords: Quality of data, task simplification, response styles, satisficing

Papers

Numbers Matter: How Numerical Scale Labels Influence Responses in a Cross-National Survey

Dr Gianmaria Bottoni (City St George's, University of London) - Presenting Author
Dr Eva Aizpurua (National Centre for Social Research (NatCen))

Survey responses are shaped by both explicit design choices and subtle features, such as the numerical labels of response scales. This study investigates the effect of numerical scale labels on response behaviour using 11-point bipolar scales with verbal anchors ("extremely negative" to "extremely positive") and two numerical label ranges: 0 to 10 and -5 to +5. Conducted in Great Britain, Hungary, and Portugal as part of a face-to-face cross-national survey, respondents were randomly assigned to one of these two scales to evaluate perceptions of climate change's overall impact and its effects on specific groups. Contrary to established findings (e.g., Schwarz et al., 1991; Tourangeau et al., 2007), mean scores were consistently lower for the -5 to +5 scale across countries and items. This suggests that numerical scale labels interact with contextual and item-specific factors, influencing response distributions in ways that may not generalise across studies. Additionally, respondents assigned to the 0 to 10 scale were more likely to select the midpoint (5) than those using the -5 to +5 scale (0), likely due to differing perceptions of neutrality associated with these options. These findings raise important questions about the context-dependence of numerical labels, particularly for polarising topics like climate change. The implications for survey design are relevant, highlighting how seemingly secondary design elements can shape data outcomes. We also discuss how some of these results differ from prior research and outline directions for future studies to better understand the nuanced effects of numerical labels across diverse topics.

The Effects of Outliers in Survey Data

Professor Jörg Blasius (University of Bonn) - Presenting Author
Professor Susanne Vogl (University of Stuttgart)

Outliers are common in all social surveys, and it is well-known that they can adversely affect the solutions of multivariate data analysis. This has been discussed especially for regression analysis, but also for principal component analysis and other scaling and cluster methods. There are many different techniques suggested in the literature for reducing the effects of outliers. In the simplest case, the affected cases are deleted. In the social sciences, outliers are rarely discussed, least of all in combination with respondent behavior such as satisficing. Further, almost no attention has been paid to those cases where the effect of outliers is ampli-fied by routinely applied techniques, e.g., rotation in principal component analysis.
In principal component analysis, categorical principal component analysis, and factor analy-sis, varimax rotation is often applied, sometimes even without discussing the unrotated solu-tion. In this paper, we show that this kind of rotation sometimes only optimally adapts the outliers of a survey, which might be caused by a small number of respondents giving arbi-trary answers or using some kind of simplified response style; this response behavior is often called satisficing. As a result, the content of the solution can be changed. To illustrate our findings, we use empirical data from a self-administered online survey of pupils aged 14–16 from lower track secondary schools in Vienna, Austria, in 2018 (N=3,078).

Estimating reliability in cross-national longitudinal survey measures

Dr Alexandru Cernat (University of Manchester) - Presenting Author
Dr Chris Antoun (University of Maryland)

Reliability in survey measurement is a crucial aspect of data quality. Yet surprisingly little attention has been given to assessing reliability for commonly used survey measures or evaluating how reliability may vary across different survey contexts. This study leverages the Comparative Panel File (CPF), a dataset compiled from long-running household panel surveys in seven countries—Australia, Germany, Russia, South Korea, Switzerland, the United Kingdom, and the United States—to estimate the reliability of 14 survey indicators from 2001 to 2020 using quasi-simplex models. We find that reliability is high and consistent across countries for factual items but substantially lower and more variable for self-assessment items (e.g., satisfaction with life, satisfaction with work, self-rated health). Additionally, we find that variations in question wording across the panels (e.g., asking about health currently versus in general) led to different reliabilities. We discuss the implications these findings have for measurement comparability in cross-national research.

Panel conditioning in measuring financial behaviour and expectations

Dr Alexandru Cernat (University of Manchester) - Presenting Author
Dr Joe Sakshaug (IAB)
Dr Bella Struminskaya (Utrecht University)
Dr Susanne Helmschrott (Bundesbank)
Dr Schmidt Tobias (Budensbank)

This study investigates the phenomenon of panel conditioning and its impact on the reliability and stability of financial behaviour and expectation measures collected through a high-frequency longitudinal survey. Panel conditioning occurs when repeated participation in a survey influences respondents' behaviours and responses, potentially leading to biased data. Utilizing a quasi-experimental design and structural equation modelling, we analyse data from a monthly online survey to assess the extent to which panel participation affects data quality. Our findings indicate that while panel conditioning can lead to a slight increase in the reliability of certain measures over time, its effects on stability are more complex and variable-dependent. This research contributes to the broader understanding of panel conditioning effects, offering methodological guidance for future studies in this domain.

Measurement Error When Surveying Democratic Support: A MultiTrait MultiError Approach

Mr Kim Backström (Åbo Akademi University) - Presenting Author
Dr Alexandru Cernat (The University of Manchester)
Dr Inga Saikkonen (Åbo Akademi University)
Professor Kim Strandberg (Åbo Akademi University)

We live in a time of political polarization and democratic backsliding. To better understand such processes, it is essential to measure such complex concepts correctly. Attitudinal survey questions, which are typically used to collect data in this area, are known to be affected by measurement error such as social desirability, acquiescence, and random error. This study Investigates the effect of different types of measurement error on survey items measuring democratic support using the MultiTrait MultiError (MTME) approach, a within-person experimental design.

The study is based on two waves of the Finnish Citizens’ Opinion panel’s municipal and regional election study and estimates concurrently correlated errors (social desirability bias, acquiescence bias, method effects) and random error. The design manipulates question-wording (positive vs. negative), response scale direction (agree-first vs. disagree-first), and scale length (5 vs. 7 points) across two measurement points, while employing latent variable modeling to estimate the errors.

We also investigate differences across key groups, such as those with lower panel recruitment propensities and those inhabiting the characteristics of a ‘critical citizen.’ Finally, we will also run sensitivity analyses to determine the occurrence of memory effects and the robustness of the social desirability estimates using proxies such as the Marlow-Crowne scale and the Big 5 personality traits.

By concurrently identifying and correcting for different types of measurement error, this study contributes to a deeper understanding of democratic support and its measurement, offering insights for improving survey research. It can also lead to better question wording and response scale selection recommendations.