ESRA 2025 Preliminary Program
All time references are in CEST
Good, fast, cheap: Pick two – Optimizing Sampling Strategies for Modern Survey Research 2 |
| Session Organisers |
Professor Sabine Zinn (DIW-SOEP / Humboldt University Berlin)
Dr Hans Walter Steinhauer (Socio-Economic Panel at DIW)
|
| Time | Wednesday 16 July, 16:00 - 17:30 |
| Room |
Ruppert A - 0.21 |
Survey research is increasingly adapting to the demands of fast-paced environments where timely, reliable data is crucial, often within limited budgets. To meet these demands, researchers frequently use non-random sampling and online data collection, which provide quick results but may lack reliability. Traditional methods that ensure accuracy are slower and more costly, yet essential for scientific research and policymaking.
This session invites contributions on the practical use of sampling frames for generating random samples, such as population registers or geo-referenced databases. We are also interested in research on non-probability sampling methods, including data from social media, affordable access panels like Prolific, and respondent-driven sampling schemes. Our goal is to examine the pros and cons of these sampling strategies, focusing on coverage, bias, generalizability, cost, and speed.
We seek discussions on optimal sampling approaches tailored to specific study needs, where researchers must balance the urgency of obtaining rapid results with the need for high-quality studies that can inform policy recommendations.
We invite submissions on:
- Innovative sampling frames for social science surveys
- Combining different sampling frames to enhance data quality and timeliness
- Methods for improving accuracy and quick data access
- Cost analyses of various sampling strategies
- Experiences using fast-access data from web providers like Prolific and Respondi for social science research
Through these discussions, we aim to guide the development of more effective and efficient approaches to survey research in today’s fast-paced data environment.
Keywords: random sampling, non-random sampling, combining sampling frames
Papers
Dial-Up or Mail-Out: The Great German Survey Showdown
Dr Matthias Sand (GESIS - Leibniz-Institut für Sozialwissenschaften) - Presenting Author
Mr Björn Rohr (GESIS - Leibniz-Institut für Sozialwissenschaften)
Professor Reinhard Pollak (GESIS - Leibniz-Institut für Sozialwissenschaften; Universität Mannheim)
This study addresses the trade-offs between cost, speed, and data quality in conducting probabilistic surveys in Germany. CATI has long been viewed as a relatively inexpensive and rapid means of data collection. However, even nationwide dual-frame CATI surveys, incorporating both mobile and landline numbers, face persistently low response rates and minimal insight into potential nonresponse biases. In case of regional CATI surveys, the problem may even be worse, since they heavily rely on landline telephone numbers, resulting in significant undercoverage and raising questions about the quality of such samples. These concerns are not only methodological but also practical, as better sampling frames and response rates may come at higher costs and longer field times.
Register-based surveys, though more time-consuming due to address acquisition, offer better coverage, data quality, and more accurate population representation. However, direct comparisons of their financial and qualitative aspects remain limited.
This research aims to fill this gap by directly comparing two samples drawn for the “Deutschland-Monitor” survey in 2024. Utilizing an identical questionnaire and simultaneous fieldwork period, one sample will be collected via a CATI survey and the other via a register-based sample. The study will not only evaluate the overall costs and data quality of these approaches but also examine the cost implications of conducting register-based surveys when it is managed in-house rather than outsourced to a fieldwork agency. This parallel setup allows for a systematic assessment of trade-offs between efficiency, cost, and data quality. The study seeks to inform researchers about the practical and methodological considerations of different survey modes. The findings will help clarify to what extent potentially more expensive, time-intensive register-based surveys may yield data of superior quality compared to faster, cheaper, but potentially more biased CATI surveys.
Understanding Firm Dynamics with Daily Data
Professor Davud Rostam-Afschar (University of Mannheim) - Presenting Author
Mr Lukas Hack (University of Mannheim)
This study introduces a novel approach to understanding firm dynamics through the German Business Panel (GBP), a high-frequency survey of German firms conducted daily over three years. The survey employs a randomized daily invitation system, ensuring a stable and representative cross-sectional composition of firms. This design enables the construction of robust daily time series, which provide unique insights into firms' plans and expectations in response to macroeconomic shocks.
The GBP survey design leverages a rolling invitation mechanism, randomly inviting subsets of firms each working day. This approach ensures steady participation across days, with response numbers averaging 45 per day, increasing to 61 on weekdays. The survey data are validated by comparing them against well-established monthly economic indicators, demonstrating strong correlations with metrics such as industrial production and inflation.
The survey questions cover key aspects of firm decision-making, including sales price plans, fixed costs, R&D investments, and dividends, as well as expectations related to revenue, profit, employment, and investment. Responses are aggregated into daily time series, weighted by the number of participating firms, allowing the measurement of high-frequency variations. The randomization and design mitigate biases associated with monthly or quarterly survey cycles and enable the exploration of real-time firm responses to economic and geopolitical events.
This methodological innovation contributes to the field by addressing limitations of traditional long-sample time series approaches, offering a robust framework for capturing firms' short-term dynamics under varying economic conditions. The GBP Daily Business Database provides a reliable and scalable foundation for further research into the effects of macroeconomic shocks, central bank policies, and other external factors on firm behavior.
BETTERSURVEYS: An Online Platform for Bias Adjustment in Non-Probability Surveys
Dr Maria del Mar Rueda García (Universidad de Granada)
Dr Andrés Cabrera-León (Universidad de Granada)
Dr Luis Castro Martín (Universidad de Granada)
Dr Beatriz Cobo Rodríguez (Universidad de Granada) - Presenting Author
Dr Ramón Ferri (Universidad de Granada)
Dr Carmen Sánchez-Cantalejo (Escuela Andaluza de Salud Pública)
Mr Jorge Luis Rueda (Universidad de Granada)
Survey methodology is currently facing significant challenges due to social and technological changes. These challenges have led to a substantial increase in refusals to participate and difficulties in accessing individuals to be interviewed, which introduce notable biases and compromise the validity of the results.
To tackle these issues, a technology platform has been developed and validated to correct biases and enhance the validity and precision of survey estimates. This platform applies advanced methodologies to infer parameters using data from diverse sources, including probability samples, non-probability samples, and administrative records. It addresses biases caused by non-sampling errors, such as self-selection, coverage, and non-response, while leveraging machine learning techniques for classification and regression.
The platform, available for free, provides adjusted weights for each sampling unit in the database, enabling researchers to conduct statistical analyses that better represent the population under study and deliver bias-adjusted parameter estimates.
Low Price, Low Information Quality? Comparison of a Random and Convenience Sample in a Municipal Survey
Dr Basha Vicari (Institute for Employment Research & Statistical Office of the City of Erlangen) - Presenting Author
The cost of high-quality surveys based on random samples is an increasing problem in social sciences. However, this problem is far more pronounced in municipal statistical offices. As decision-makers expect cost-effective and rapid findings on specific issues, citizen surveys are conducted and instantly used for policy advice. For this reason, municipal statisticians often use convenience samples. However, the validity and representativeness of such convenience samples is questionable.
In a German metropolitan municipality, we were able to conduct a survey based on a random sample and a subsequent non-random survey. In the first part of the survey on satisfaction with waste management, cleanliness in the city, and attitudes towards environmental protection measures 3,000 randomly selected citizens from the population register were invited by mail to participate in the hybrid survey (paper or online questionnaire). The response rate was 31 percent.
In the second part, which began a few weeks after the first part was completed, participants were only recruited via various advertising measures. These include the city's homepage and social media channels, an article in the local newspaper, digital billboards on major streets, and posters at waste collection points inviting people to participate via a link or QR code over four weeks.
After completing the second part of the survey, we will be able to compare both types of samples in terms of coverage, bias, generalizability, and cost. This comparison will provide insights into the use and limitations of cheap convenience samples. We also intend to test the assumption that it-affine population groups, such as young men without a migration background, participate over-proportionally in non-random surveys.
