ESRA 2025 Preliminary Glance Program

All time references are in CEST

Quality Assurance in Survey Data: Frameworks, Tools, and Quality Indicators 3
Session Organisers	Dr Jessica Daikeler (GESIS- Leibniz Institute for the Social Sciences ) Mrs Fabienne Krämer (GESIS- Leibniz Institute for the Social Sciences )
Time	Wednesday 16 July, 09:00 - 10:30
Room	Ruppert 114

Survey data, collected through various modes such as online surveys, face-to-face interviews, and telephone surveys, is subject to a wide range of potential errors that can compromise data integrity. Addressing these challenges requires robust frameworks, advanced tools, and reliable quality indicators to manage, validate, and enhance survey data quality.
This session will focus on the key aspects of quality assurance in survey data collection and analysis, with a particular emphasis on the development and application of data quality indicators. We invite contributions that suggest and showcase quality indicators, designed to maintain the integrity and usability of survey data. Key topics will include:
1. Frameworks for Quality Assurance: An overview of frameworks developed to assess the quality of survey data.
2. Tools and Platforms for Data Validation: A discussion on tools and technologies aimed at validating or improving the quality of survey data as well as platforms tailored to combine tools, such as the KODAQS toolbox.
3. Data quality indicators: We seek contributions that demonstrate effective use of quality indicators like response bias indicators or data consistency checks in real-world case studies, showcasing how they address and enhance data quality.
4. Didactics of Data Quality Issues: Approaches to teaching and promoting data quality assurance for survey data. This section will explore educational strategies to equip researchers and practitioners with the necessary skills to effectively tackle data quality issues.

Keywords: survey quality tools, data quality, frameworks, quality indicators, training, didactics

Papers

Let’s Talk About Limitations: Analyzing Data Quality Aspects Mentioned in Limitation Sections

Dr Fiona Draxler (University of Mannheim) - Presenting Author

Data quality frameworks and reporting guidelines support researchers in identifying potential data quality concerns with their research. However, it is unclear to what extent publications actually report on data quality limitations and which aspects are rarely mentioned.

We analyze the “Limitations” sections and limitation paragraphs in “Discussion” sections of substantive survey-based research published in selected journals including Public Opinion Quarterly, the American Sociological Review, and the Annual Review of Sociology. We extract data-quality-related keywords of these paragraphs/sections and cluster them by themes, in alignment with the components of the Total Survey Error framework.

Based on this, we discuss what data quality limitations are commonly and rarely mentioned, and what possible reasons for these differences may be. Through comparisons with reporting guidelines such as those of the AAPOR transparency initiative, we highlight areas where researchers might require additional support in choosing suitable identifiers and reporting quality limitations. We also analyze areas where current guidelines might be adapted to better represent researchers’ needs in reporting. Thus, we contribute to the transparent and well-structured communication of data quality as a crucial step for validating research.

Approaches to teaching and promoting data quality assurance and sharing for longitudinal population studies

Ms Jo Webb (UK Data Service, UK Data Archive, University of Essex)
Mrs Cristina Magder (UK Data Service, UK Data Archive, University of Essex)
Dr Sharon Bolton (UK Data Service, UK Data Archive, University of Essex)
Ms Liz Smy (UK Data Service, UK Data Archive, University of Essex)
Dr Hina Zahid (UK Data Service, UK Data Archive, University of Essex)
Mrs Beate Lichtwardt (UK Data Service, UK Data Archive, University of Essex) - Presenting Author
Mrs Gail Howell (UK Data Service, UK Data Archive, University of Essex)
Dr Finn Dymond-Green (UK Data Service, JISC)

This presentation explores strategies to equip data managers and practitioners with the skills needed to address data quality challenges effectively. Drawing on the “Skills Development for Managing Longitudinal Data for Sharing” project, commissioned by the ESRC and MRC as part of initial Population Research UK (PRUK) efforts, we present insights and resources developed to enhance data management and sharing practices within the Longitudinal Population Studies (LPS) community. This initiative responds to critical challenges UK Research and Innovation (UKRI) identified to maximise the use of LPS data across social, economic, and biomedical sciences.

Our presentation highlights the interactive training workshops designed and delivered to over 300 data managers and professionals. These workshops emphasised foundational to advanced skills, such as synthetic data creation and harmonisation tools and incorporated continuous evaluation to adapt to the community's evolving needs. The freely available, open-licensed training materials developed during this project provide a practical resource to support the LPS community and broader data professionals in improving data quality assurance and sharing, aligning with PRUK’s vision for impactful data use.

Assessing response quality in surveys: A systematic review of the use and interpretation of quality indicators

Ms Sophia Piesch (University of Mannheim) - Presenting Author
Professor Florian Keusch (University of Mannheim)

In the literature, various indicators such as straightlining and acquiescence have been proposed to detect non-optimal response behavior and assess the quality of survey data. One of the key challenges in effectively using and interpreting these indicators is the inconsistency in terminology, theoretical concepts, and methods for constructing them. While survey methodology literature predominantly examines these indicators within the framework of satisficing theory (Krosnick, 1991, 1999), psychological research draws on the concept of response styles (Paulhus, 1991). These concepts partly overlap in their theoretical assumptions, however, are defined differently and assume different underlying cognitive processes. Yet, researchers frequently use them interchangeably and imprecisely, both within and across disciplines. This general lack of clarity hinders the comparability of studies and utility of these indicators to assess the quality of responses.
To address this issue, we conduct a comprehensive, multidisciplinary systematic review of empirical studies published since 2010 on response quality indicators. We provide a structured overview of how researchers measure and conceptualize different response quality indicators and document how these indicators vary as a function of personal, situational, instrument-related characteristics. Our review will allow us to address key questions about the theoretical relationships between these indicators: Can findings related to one indicator be generalized to others? Do the different indicators reflect the same underlying construct, or are they influenced by different factors? A particular strength of our review is that we widely search for evidence across different strands of literature, rather than focusing on either satisficing theory or response styles, which have been the subject of previous reviews (Roberts et al., 2019; Van Vaerenbergh & Thomas, 2013). By doing so, we aim to leverage insights from different research areas to gain a better conceptual understanding of these indicators and to inform researchers on how to properly measure and apply them.

A Risk-Based Approach to Mitigating Online Survey Fraud

Dr Phillips Benjamin (Social Research Centre) - Presenting Author
Dr Dina Neiger (Social Research Centre)
Mr Kipling Zubevich (Social Research Centre)

Fraud is rife in samples recruited via open links (c.f. Bonett et al. 2024; Johnson et al. 2024; Keeter et al. 2024; Pinzón et al. n.d.; White-Cascarilla and Brodhead n.d.). However, even online surveys protected using unique URLs or login pages are at risk from fraudsters using mass attacks to guess login information and obtain incentives. Defeating such attacks requires countermeasures that can negatively impact the respondent experience—at a time when response rates already at perilously low levels and serious nonresponse error is common—and will give rise to false positives for fraud, where these countermeasures include requiring respondents to pass CAPTCHAs, use of digital fingerprinting software to identify duplicate and fraudulent responses, and offline payment of incentives. Ideally, the severity of countermeasures to online survey fraud should be proportional to the risk, accounting for the likelihood and severity of impact of fraud.
We present a risk-based approach to determining an appropriate level of mitigation of fraud risk, balancing the strength of mitigations against the degree of risk. On the risk side, the approach accounts for factors including whether PII will be piped in (e.g., from prior waves or the sampling frame), the nature of the sampling frame and means of recruitment, the incentives used, the number of invitations, and the uses to which the research will be put (e.g., as input to public policy decisions). Mitigations considered include CAPTCHAs, digital fingerprinting and other fraud detection software, two factor authentication, user ID complexity, means of incentive payment, and planned QC activities. The two are balanced using a set of importance weights, which have been refined through use of the tool.
This paper contributes to the ongoing development of approaches to address fraud in online surveys.