ESRA 2025 Preliminary Program
All time references are in CEST
Good, fast, cheap: Pick two – Optimizing Sampling Strategies for Modern Survey Research |
| Session Organisers |
Professor Sabine Zinn (DIW-SOEP / Humboldt University Berlin)
Dr Hans Walter Steinhauer (Socio-Economic Panel at DIW)
|
| Time | Wednesday 16 July, 13:45 - 15:00 |
| Room |
Ruppert A - 0.21 |
Survey research is increasingly adapting to the demands of fast-paced environments where timely, reliable data is crucial, often within limited budgets. To meet these demands, researchers frequently use non-random sampling and online data collection, which provide quick results but may lack reliability. Traditional methods that ensure accuracy are slower and more costly, yet essential for scientific research and policymaking.
This session invites contributions on the practical use of sampling frames for generating random samples, such as population registers or geo-referenced databases. We are also interested in research on non-probability sampling methods, including data from social media, affordable access panels like Prolific, and respondent-driven sampling schemes. Our goal is to examine the pros and cons of these sampling strategies, focusing on coverage, bias, generalizability, cost, and speed.
We seek discussions on optimal sampling approaches tailored to specific study needs, where researchers must balance the urgency of obtaining rapid results with the need for high-quality studies that can inform policy recommendations.
We invite submissions on:
- Innovative sampling frames for social science surveys
- Combining different sampling frames to enhance data quality and timeliness
- Methods for improving accuracy and quick data access
- Cost analyses of various sampling strategies
- Experiences using fast-access data from web providers like Prolific and Respondi for social science research
Through these discussions, we aim to guide the development of more effective and efficient approaches to survey research in today’s fast-paced data environment.
Keywords: random sampling, non-random sampling, combining sampling frames
Papers
Experience sampling method supported by temperature and humidity data
Dr Marcin Witkowski (Adam Mickiewicz University) - Presenting Author
Mr Paweł Felcyn (Adam Mickiewicz University)
Professor Piotr Jabkowski (Adam Mickiewicz University)
Collecting survey data to assess parameters that are subjective and varies within a population presents challenges when there is no objective source for their evaluation. One such parameter is the assessment of thermal comfort based on temperature and humidity in the environment of surveyed individuals. Temperature data are either available at a very general aggregation level (e.g., several weather station delivering measurements points in a city) or as satellite data, which do not accurately reflect indoor conditions, low altitudes, humidity, how urban greenery and water bodies can mitigate the heat effect etc.
As part of the UrbEaT project (https://urbeat.site/), we are conducting research on the urban heat island effect and methods for its mitigation. The studies are being carried out as comparative study of Beijing and Warsaw. For the project, we developed an open source mobile application called GeoSenEsm, which allows for collecting survey data together with data from Xiaomi and Kestrel temperature and humidity sensors. During the summer of 2025, over 200 study participants will use those sensors to measure temperature and humidity in their surroundings.
In our presentation, we will present the application, discuss issues related to connectivity and data collection, the impact of various factors on data readings as well as the results of the ongoing survey conducted using the experience sampling method.
High-Quality, Fast, and Cheap? Nonresponse, Selection Bias, and Survey Costs in Probability-Based High-Frequency Online Panels
Dr Mustafa Coban (Institute for Employment Research (IAB)) - Presenting Author
The COVID-19 pandemic has accelerated the ongoing shift in survey research toward self-administered online data collection. Recently, high-quality web surveys have achieved response rates comparable to, or even exceed, telephone surveys. However, selection bias and panel attrition often emerge more prominently in online panels. These challenges are especially critical in web surveys with extensive questionnaire programs or higher survey frequencies, as they can increase participant burden and dropout rates.
In 2023, the Institute for Employment Research (IAB) in Germany launched a new online survey, the IAB-Online Panel for Labour Market Research (IAB-OPAL), a panel survey of the German labour force aged between 18 and 65. The IAB initiated this quarterly survey using a push-to-web approach. Addresses were sampled from a comprehensive database that includes compulsory social insurance notifications from employers, as well as unemployment insurance and welfare benefit records.
This unique opportunity allows for the use of a sampling frame with detailed individual-level data from complete employment histories to address three central research questions:
(1) How do the different stages of the recruitment process for an online panel contribute to nonresponse and selection bias?
(2) How do selection bias and attrition evolve over time in a high-frequency online panel survey?
(3) What are the survey costs associated with recruitment and panel stability?
Using detailed individual-level data from complete employment biographies allows for precise analysis of how biases and nonresponse emerge throughout the recruitment process. Additionally, it facilitates tracking the evolution of selection bias and panel attrition across socioeconomic and demographic groups over successive survey waves. Finally, I calculate average costs for recruiting one-time respondents and one-year panelists, providing valuable insights into cost-efficiency by demographic and socioeconomic profiles.
Population Research with Non-Population Data? The Accuracy of Findings on Self-Rated Health in U.S. Non-Probability Surveys
Dr Liliya Leopold (University of Amsterdam) - Presenting Author
Mrs Noble Nolen (University of Amsterdam)
Mr Diego Strassmann Rocha (University of Bremen)
Dr Thomas Leopold (University of Cologne)
Dr Brian O'Shea (University of Nottingham)
This study assesses the accuracy of findings on self-rated health (SRH) and its demographic variations across gender, education, ethnicity, race, and age in non-probability surveys, evaluating their potential to advance research on health disparities.
We analyzed three non-probability surveys with extensive health measures: the Research and Development Survey 8 (RANDS-8), the Quota-based Population Health Survey (QPHS), and the Project Implicit – Health Survey (PI-H). These surveys were benchmarked against the U.S. Census’s Current Population Survey (CPS) and compared with four probability-based surveys: the National Health Interview Survey (NHIS), National Health and Nutrition Examination Survey (NHANES), General Social Survey (GSS), and the Understanding America Study (UAS). Data from 141,008 respondents aged 18 to 65, collected between 2022 and 2024, were included.
While non-probability surveys generally exhibited lower accuracy than probability-based surveys, univariate SRH distributions in RANDS-8 and QPHS, along with certain multivariate estimates—such as age-SRH and education-SRH associations in RANDS-8 and QPHS, and Black race-SRH associations in PI-H—showed strong alignment with the benchmark. Conversely, gender-SRH associations in all non-probability surveys substantially overstated effect sizes compared to benchmarks, while ethno-racial disparities in SRH in RANDS-8 and QPHS diverged strongly, even reversing direction.
These findings underscore both the promise and the challenges of using non-probability surveys to study SRH and its demographic variations. They highlight the need for rigorous validation against reliable benchmarks to ensure accurate insights into health disparities.
