ESRA 2025 Preliminary Program
All time references are in CEST
Quality Assurance in Survey Data: Frameworks, Tools, and Quality Indicators 2 |
| Session Organisers |
Dr Jessica Daikeler (GESIS- Leibniz Institute for the Social Sciences )
Mrs Fabienne Krämer (GESIS- Leibniz Institute for the Social Sciences )
|
| Time | Tuesday 15 July, 14:00 - 15:00 |
| Room |
Ruppert 134 |
Survey data, collected through various modes such as online surveys, face-to-face interviews, and telephone surveys, is subject to a wide range of potential errors that can compromise data integrity. Addressing these challenges requires robust frameworks, advanced tools, and reliable quality indicators to manage, validate, and enhance survey data quality.
This session will focus on the key aspects of quality assurance in survey data collection and analysis, with a particular emphasis on the development and application of data quality indicators. We invite contributions that suggest and showcase quality indicators, designed to maintain the integrity and usability of survey data. Key topics will include:
1. Frameworks for Quality Assurance: An overview of frameworks developed to assess the quality of survey data.
2. Tools and Platforms for Data Validation: A discussion on tools and technologies aimed at validating or improving the quality of survey data as well as platforms tailored to combine tools, such as the KODAQS toolbox.
3. Data quality indicators: We seek contributions that demonstrate effective use of quality indicators like response bias indicators or data consistency checks in real-world case studies, showcasing how they address and enhance data quality.
4. Didactics of Data Quality Issues: Approaches to teaching and promoting data quality assurance for survey data. This section will explore educational strategies to equip researchers and practitioners with the necessary skills to effectively tackle data quality issues.
Keywords: survey quality tools, data quality, frameworks, quality indicators, training, didactics
Papers
Assessing Data Quality in Opt-In Versus Probability-Based Online Panels in Europe Using KnowledgePanel Europe
Mrs Cristina Tudose (Ipsos KnowledgePanel Europe) - Presenting Author
Dr Joke Depraetere (Ipsos KnowledgePanel Europe)
Dr Femke Dekeulenaer (Ipsos KnowledgePanel Europe)
This study delves into the critical issue of data quality variations between opt-in and probability-based online panels, a topic of increasing relevance in online survey research. This study analyzes such variations using four sample sources across Sweden, France, and the Netherlands. Each set includes a probability-based sample from Ipsos’ KnowledgePanel Europe and three opt-in samples.
The study evaluates how effectively demographic structures are met and compares the prevalence of low quality across several sample sources. Low quality responses can compromise data reliability and lead to flawed conclusions. Both standard quality metrics (e.g., speeding, straight-lining) and specific checks for questionnaire inconsistencies as an additional quality tool are used to evaluate quality of responses. The analysis also incorporates an investigation of online survey behavior and answer patterns in open-ended questions, offering insights into response quality across sample sources. By exploring the interplay between survey frequency and survey professionalization, the study sheds light on potential biases and their influence on data quality, further enriching the understanding of the factors contributing to data quality variations.
This research provides insight into the comparative quality of opt-in and probability-based samples and the variations of quality within opt-in samples, informing researchers and practitioners on appropriate survey methodologies for European contexts. The study's findings contribute to understanding the strengths and limitations of different online sampling approaches, ultimately enhancing data quality and research reliability.
Comparative quality indicators for capturing latent constructs with single questions
Dr Ranjit K. Singh (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
It is challenging to assess the quality of single questions for latent constructs, such as values, opinions, or interests. Psychometry instead uses multiple indicators, which allow us to judge measurement quality with factor analyses and measures of internal consistency. However, for single questions we cannot rely on the same methods.
I will argue that we can sidestep some of these limitations by comparing different single questions for the same construct. I transfer a construct validation approach proposed by Westen and Rosenthal to the realm of single-item measures in social science surveys. Building upon research on ex-post harmonization, I show that we can learn much about either question by comparing how they both correlate to the set of covariates. The method gives us a metric of how similar the constructs are that both questions measure and a coefficient that quantifies the relative reliabilities of both questions. Note that we do not need answers to both questions from the same set of respondents. We merely need two independent random samples from the same population; one for each question. A structure akin to split-ballot experiments.
The talk extends research presented at the last ESRA conference by exploring the two metrics with a set of simulations, a larger survey data set, and by applying it to data quality assessment (instead of data integration). Aside from demonstrating the validity of both metrics, I aim to show pragmatic use cases. For example, demonstrating empirically that we can use quite different questions (e.g., interest in political TV shows) as serviceable proxies for some constructs (e.g., general political interest). I also show that if we have quantified the reliability of one question, we can leverage that information to predict the quality of another question on the same construct.
Measuring and Improving the Quality of Open-Ended Survey Responses
Dr Masha Krupenkin (University of Maryland) - Presenting Author
Dr Andrew Gordon (Prolific)
Dr David Rothschild (Microsoft Research)
With the recent proliferation of automated text analysis methods, scholars are increasingly turning to open-ended survey responses as a measure of public opinion. However, the quality of open-ended responses on online surveys can be extremely variable. This paper presents several methods to assess and improve the quality of open-ended survey responses.
We develop and test four measures to determine the quality of open-ended responses. In Measure 1 (baseline), we use answer wordcount as a baseline measure of answer quality. In Measure 2 (human-coded), we employ workers on Prolific to rate open-ended survey responses. In Measure 3 (GPT), we train a GPT-4 model to rate OE responses using the training set of human ratings generated for Measure 2. In Measure 4 (Hybrid), we use human annotators to assess and adjust the LLM ratings generated by GPT-4.
We examine variability in OE data quality based on several sets of respondent characteristics. The first, platform, is a proxy for general audience quality. Our second set of respondent characteristics examines prior platform experiences, including number of studies taken, approval rate, and more. Finally, we examine how respondent age and education shapes OE answer quality.
We also test the impact of survey infrastructure on OE data quality. We test the effects of desktop vs mobile infrastructure, as well as text vs voice answer options. We also examine the interaction between infrastructure and audience characteristics. Does more accessible OE infrastructure lead to greater increases in OE answer quality for specific audiences?
Finally, we examine the sensitivity of different summary models to OE response quality. We assess the following summary models: Topic Models, BERT, GPT-4. In addition to testing these models on real OE responses, we test these models using a synthetic dataset generated by GPT-4 that
