ESRA 2025 Preliminary Program
All time references are in CEST
Measurement errors in smart surveys: Accomplishments, challenges, and looking forward |
| Session Organisers |
Dr Jonas Klingwort (Statistics Netherlands (CBS) - Department of Research & Development)
Dr Vera Toepoel (Statistics Netherlands (CBS) - Department of Research & Development)
Professor Peter Lugtig (Utrecht University - Methodology and Statistics)
|
| Time | Tuesday 15 July, 11:00 - 12:00 |
| Room |
Ruppert 134 |
Integrating digital technologies into survey research transforms the survey landscape and leads to the development of smart surveys, which have at least one of the following features: internal/external sensors, open/public online data, personal online data, or linkage consent. This session is dedicated to accomplishments in smart surveys, focusing on using sensors, mobile apps, and data donation. These innovations offer opportunities to improve data quality, reduce respondent burden, and capture real-time behavior. However, they also bring challenges, particularly concerning measurement errors. This session focuses on the accomplishments and challenges associated with these technologies, emphasizing understanding, quantifying, and mitigating measurement errors.
Sensors in smartphones, wearables, and connected devices collect passive data on physical activity, environmental conditions, or physiological responses. Despite their potential, sensors introduce measurement errors due to calibration issues, data loss, and variability in sensor sensitivity. This session will discuss how researchers address these errors to ensure reliable and valid data collection.
Mobile apps have emerged as tools to engage survey participation and enable fine-granular, real-time data collection. However, apps can also introduce measurement errors, including biases related to device compatibility, app design, and respondent behavior. We want to discuss how these factors impact data quality and what strategies are being implemented to reduce measurement error in app-based survey research.
Data donation, where participants voluntarily share digital traces---such as social media activity, browsing history, or transactional data---offers an opportunity to gain insights into behaviors and attitudes. However, this approach suffers from self-selection bias, incomplete data, and varying levels of data accuracy. We want to address how these measurement errors can be mitigated through careful study design and respondent recruitment strategies.
We invite researchers, practitioners, and technologists to discuss harnessing these innovations while highlighting the complexities they introduce to the survey research landscape.
Keywords: sensors, mobile apps, data donation, GPS data, digital traces, passive data collection, survey errors
Papers
Collecting Smart Meter Data To Gain Insights In The Energy Consumption Of Households
Mrs Janelle van den Heuvel (Statistics Netherlands) - Presenting Author
Dr Maaike Kompier (Statistics Netherlands)
Mr Jeldrik Bakker (Statistics Netherlands)
Climate change is a big world-wide challenge. Statistics about energy consumption are important sources of information based on which policies can be adapted and with which effectiveness of certain energy saving measures can be assessed. Currently, Statistics Netherlands publishes its energy statistics on a yearly basis and statistics about energy behavior even only once every 6 years. More detailed insights in energy consumption is important to meet Eurostat and government requirements. The advent of smart meters provides ample opportunities for detailed measurement of energy consumption. The meters provide high frequency data (hourly, or even quarterly) with which the current statistics can be improved and extended.
There are multiple ways to collect smart meter data. Respondents can access and export data through (their own energy providers’) online portals, or by plugging a dongle into the P1 port of their meter. What method would be most feasible resulting in high response and representativity is yet u
Detection of “Sloppy” Respondents in a Web Survey
Mrs Diane Maillot-Tchofo (Médiamétrie) - Presenting Author
Mr Tom Devynck (Toulouse School of Economics)
Mrs Fabienne Le Sager (Médiamétrie)
Mr Louis Marec (Médiamétrie)
The most familiar observational error within surveys is associated with respondents’ inability or unwillingness to provide the correct answer. In this context, the will to correctly estimate digital devices ownership (e.g. TV, smartphone) drove Médiamétrie (France) to develop an ambivalent method to detect neglectful respondents in a web survey (CAWI). One of the main challenges of this study lies in the nature of the data which is an already-in-production survey.
We drew from Laura Gamble (2023) and Anvita Mahajan (2023) works and derived an ambivalent method combining both approaches. Our first approach makes use of the questionnaire’s completion times. The second approach is a two-step clustering algorithm focused on the ownership of digital equipment. A K-Means algorithm was applied on the respondent’s household socio-demographic characteristics. Then, to each cluster both DBSCAN and Isolation Forest models were applied to contain the models’ shortcomings. Our final list of sloppy respondents was obtained by combining the results of the two approaches.
With no ground truth labels available, our results were assessed based on the characteristics of the alleged neglectful respondents against those of the overall study population. The dataset used to define the models contained around 8000 respondents, approximately 180 were flagged as sloppy. Another test dataset of the same size resulted in 304 sloppy respondents. Neglectful respondents’ characteristics and statistically significant deviations from the overall studied population were coherent with prior expectations and hypothesis. Indeed, under 35-year-olds and smartphones as the mean of response to the survey are overrepresented (resp. 35% against 19% & 60% against 45%), conversely, retirees are underrepresented (16.5% against 28.5%).
The conducted study enabled us to single-out neglectful respondents’ profiles, which is another step to take measures against observational errors and ensure data quality.
