All time references are in CEST
Good, fast, cheap: Pick two – Optimizing Sampling Strategies for Modern Survey Research |
|
| Session Organisers | Professor Sabine Zinn (DIW-SOEP / Humboldt University Berlin) Dr Hans Walter Steinhauer (Socio-Economic Panel at DIW) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Survey research is increasingly adapting to the demands of fast-paced environments where timely, reliable data is crucial, often within limited budgets. To meet these demands, researchers frequently use non-random sampling and online data collection, which provide quick results but may lack reliability. Traditional methods that ensure accuracy are slower and more costly, yet essential for scientific research and policymaking.
This session invites contributions on the practical use of sampling frames for generating random samples, such as population registers or geo-referenced databases. We are also interested in research on non-probability sampling methods, including data from social media, affordable access panels like Prolific, and respondent-driven sampling schemes. Our goal is to examine the pros and cons of these sampling strategies, focusing on coverage, bias, generalizability, cost, and speed.
We seek discussions on optimal sampling approaches tailored to specific study needs, where researchers must balance the urgency of obtaining rapid results with the need for high-quality studies that can inform policy recommendations.
We invite submissions on:
- Innovative sampling frames for social science surveys
- Combining different sampling frames to enhance data quality and timeliness
- Methods for improving accuracy and quick data access
- Cost analyses of various sampling strategies
- Experiences using fast-access data from web providers like Prolific and Respondi for social science research
Through these discussions, we aim to guide the development of more effective and efficient approaches to survey research in today’s fast-paced data environment.
Keywords: random sampling, non-random sampling, combining sampling frames
Dr Matthias Sand (GESIS - Leibniz-Institut für Sozialwissenschaften) - Presenting Author
Mr Björn Rohr (GESIS - Leibniz-Institut für Sozialwissenschaften)
Professor Reinhard Pollak (GESIS - Leibniz-Institut für Sozialwissenschaften; Universität Mannheim)
This study addresses the trade-offs between cost, speed, and data quality in conducting probabilistic surveys in Germany. CATI has long been viewed as a relatively inexpensive and rapid means of data collection. However, even nationwide dual-frame CATI surveys, incorporating both mobile and landline numbers, face persistently low response rates and minimal insight into potential nonresponse biases. In case of regional CATI surveys, the problem may even be worse, since they heavily rely on landline telephone numbers, resulting in significant undercoverage and raising questions about the quality of such samples. These concerns are not only methodological but also practical, as better sampling frames and response rates may come at higher costs and longer field times.
Register-based surveys, though more time-consuming due to address acquisition, offer better coverage, data quality, and more accurate population representation. However, direct comparisons of their financial and qualitative aspects remain limited.
This research aims to fill this gap by directly comparing two samples drawn for the “Deutschland-Monitor” survey in 2024. Utilizing an identical questionnaire and simultaneous fieldwork period, one sample will be collected via a CATI survey and the other via a register-based sample. The study will not only evaluate the overall costs and data quality of these approaches but also examine the cost implications of conducting register-based surveys when it is managed in-house rather than outsourced to a fieldwork agency. This parallel setup allows for a systematic assessment of trade-offs between efficiency, cost, and data quality. The study seeks to inform researchers about the practical and methodological considerations of different survey modes. The findings will help clarify to what extent potentially more expensive, time-intensive register-based surveys may yield data of superior quality compared to faster, cheaper, but potentially more biased CATI surveys.
Ms Carolyn Lau (Pew Research Center) - Presenting Author
Ms Georgina Pizzolitto (Pew Research Center)
Ms Sofi Sinozich (Pew Research Center)
Dr Patrick Moynihan (Pew Research Center)
Random-digit-dial (RDD) surveys have become less popular over the past decade or so as response rates dropped to the low single-digits and the cost and effort required to reach respondents rose significantly. Yet unlike other probability-based methods, RDD samples are widely available throughout Europe and easily accessible to researchers, offering high population coverage and the potential for relatively quick data collection. Therefore, RDD surveys can still hold an important place in the researcher’s toolkit, particularly if some of its inefficiencies are addressed.
One such inefficiency is the need to dial thousands of numbers to reach the target number of respondents. If the sample can be screened ahead of time to remove likely nonworking numbers, this can make fieldwork much more manageable. Pew Research Center’s annual Global Attitudes Project (GAP), which has traditionally used unfiltered RDD samples in Europe, provides an opportunity to test three screening methods for mobile sample:
(1) Home Location Register (HLR) lookup, which queries a mobile network operator’s database of numbers to determine their working status,
(2) Activity flags that are assigned to a number based on use of social media and messaging apps, and
(3) Silent SMS, which sends an SMS message to a mobile number without any notification on the recipient’s end and uses the delivery report to determine its working status.
These methods will be tested, as available, during the GAP 2025 cycle in France, Germany, Italy, the Netherlands, Spain, and the UK. The analysis will assess the accuracy of these methods in identifying working and nonworking numbers and examine the potential noncoverage implications of using screened sample – that is, how would sample demographics and attitudinal estimates change if the sample was restricted to the numbers flagged as working by each of the three methods?
Professor Davud Rostam-Afschar (University of Mannheim) - Presenting Author
Mr Lukas Hack (University of Mannheim)
This study introduces a novel approach to understanding firm dynamics through the German Business Panel (GBP), a high-frequency survey of German firms conducted daily over three years. The survey employs a randomized daily invitation system, ensuring a stable and representative cross-sectional composition of firms. This design enables the construction of robust daily time series, which provide unique insights into firms' plans and expectations in response to macroeconomic shocks.
The GBP survey design leverages a rolling invitation mechanism, randomly inviting subsets of firms each working day. This approach ensures steady participation across days, with response numbers averaging 45 per day, increasing to 61 on weekdays. The survey data are validated by comparing them against well-established monthly economic indicators, demonstrating strong correlations with metrics such as industrial production and inflation.
The survey questions cover key aspects of firm decision-making, including sales price plans, fixed costs, R&D investments, and dividends, as well as expectations related to revenue, profit, employment, and investment. Responses are aggregated into daily time series, weighted by the number of participating firms, allowing the measurement of high-frequency variations. The randomization and design mitigate biases associated with monthly or quarterly survey cycles and enable the exploration of real-time firm responses to economic and geopolitical events.
This methodological innovation contributes to the field by addressing limitations of traditional long-sample time series approaches, offering a robust framework for capturing firms' short-term dynamics under varying economic conditions. The GBP Daily Business Database provides a reliable and scalable foundation for further research into the effects of macroeconomic shocks, central bank policies, and other external factors on firm behavior.
Dr Maria del Mar Rueda García (Universidad de Granada)
Dr Andrés Cabrera-León (Universidad de Granada)
Dr Luis Castro Martín (Universidad de Granada)
Dr Beatriz Cobo Rodríguez (Universidad de Granada) - Presenting Author
Dr Ramón Ferri (Universidad de Granada)
Dr Carmen Sánchez-Cantalejo (Escuela Andaluza de Salud Pública)
Mr Jorge Luis Rueda (Universidad de Granada)
Survey methodology is currently facing significant challenges due to social and technological changes. These challenges have led to a substantial increase in refusals to participate and difficulties in accessing individuals to be interviewed, which introduce notable biases and compromise the validity of the results.
To tackle these issues, a technology platform has been developed and validated to correct biases and enhance the validity and precision of survey estimates. This platform applies advanced methodologies to infer parameters using data from diverse sources, including probability samples, non-probability samples, and administrative records. It addresses biases caused by non-sampling errors, such as self-selection, coverage, and non-response, while leveraging machine learning techniques for classification and regression.
The platform, available for free, provides adjusted weights for each sampling unit in the database, enabling researchers to conduct statistical analyses that better represent the population under study and deliver bias-adjusted parameter estimates.
Dr Jessica Daikeler (GESIS- Leibniz Institute for the Social Sciences )
Dr Joachim Pipienburg (GESIS- Leibniz Institute for the Social Sciences ) - Presenting Author
Dr Barbara Binder (GESIS- Leibniz Institute for the Social Sciences )
Dr Henning Silber (University of Michigan)
Recruiting survey participants has become increasingly difficult due to declining response rates. Piggybacking, which recruits participants for additional studies through existing surveys, offers a promising strategy. However, research on its effectiveness remains limited. This study examines piggybacking's potential using the ALLBUS 2023 (a cross-sectional survey) and the GESIS Panel.pop (a longitudinal survey) to recruit participants for the GESIS Panel.dbd under varying incentives and salience conditions.
Two research questions guided the study: (1) How do salience and financial incentives influence willingness to participate in piggybacking recruitment? and (2) Does piggybacking effectiveness differ between longitudinal and cross-sectional surveys? A 2x2 experimental design varied incentive levels (€5 or €10) and salience (highlighting the incentive or not). The ALLBUS sample, limited to self-administered modes, included 3,297 respondents (1,710 PAPI and 1,587 CAWI). Recruitment outcomes—agreement to future contact—were analyzed across experimental conditions. The same experiment was conducted in the full GESIS Panel.pop sample.
Preliminary ALLBUS results show marginal effects of salience and incentive levels on participation. Agreement was highest (70.2%) among CAWI respondents offered €10 with salience, but overall differences between conditions were small. Among respondents offered €5, higher salience reduced agreement by 5 percentage points (PP), while higher salience increased agreement by 5 PP for those offered €10. These findings suggest minor effects of incentives and salience, with other factors, such as initial survey characteristics, likely playing a larger role.
Results from the GESIS Panel.pop recruitment (expected in early 2025) will provide comparative insights into longitudinal survey respondents. This study underscores the potential of piggybacking and highlights the need for further research to uncover the mechanisms driving its effectiveness, advancing recruitment methodologies and informing future survey design.
Dr Basha Vicari (Institute for Employment Research & Statistical Office of the City of Erlangen) - Presenting Author
The cost of high-quality surveys based on random samples is an increasing problem in social sciences. However, this problem is far more pronounced in municipal statistical offices. As decision-makers expect cost-effective and rapid findings on specific issues, citizen surveys are conducted and instantly used for policy advice. For this reason, municipal statisticians often use convenience samples. However, the validity and representativeness of such convenience samples is questionable.
In a German metropolitan municipality, we were able to conduct a survey based on a random sample and a subsequent non-random survey. In the first part of the survey on satisfaction with waste management, cleanliness in the city, and attitudes towards environmental protection measures 3,000 randomly selected citizens from the population register were invited by mail to participate in the hybrid survey (paper or online questionnaire). The response rate was 31 percent.
In the second part, which began a few weeks after the first part was completed, participants were only recruited via various advertising measures. These include the city's homepage and social media channels, an article in the local newspaper, digital billboards on major streets, and posters at waste collection points inviting people to participate via a link or QR code over four weeks.
After completing the second part of the survey, we will be able to compare both types of samples in terms of coverage, bias, generalizability, and cost. This comparison will provide insights into the use and limitations of cheap convenience samples. We also intend to test the assumption that it-affine population groups, such as young men without a migration background, participate over-proportionally in non-random surveys.
Dr Mustafa Coban (Institute for Employment Research (IAB)) - Presenting Author
The COVID-19 pandemic has accelerated the ongoing shift in survey research toward self-administered online data collection. Recently, high-quality web surveys have achieved response rates comparable to, or even exceed, telephone surveys. However, selection bias and panel attrition often emerge more prominently in online panels. These challenges are especially critical in web surveys with extensive questionnaire programs or higher survey frequencies, as they can increase participant burden and dropout rates.
In 2023, the Institute for Employment Research (IAB) in Germany launched a new online survey, the IAB-Online Panel for Labour Market Research (IAB-OPAL), a panel survey of the German labour force aged between 18 and 65. The IAB initiated this quarterly survey using a push-to-web approach. Addresses were sampled from a comprehensive database that includes compulsory social insurance notifications from employers, as well as unemployment insurance and welfare benefit records.
This unique opportunity allows for the use of a sampling frame with detailed individual-level data from complete employment histories to address three central research questions:
(1) How do the different stages of the recruitment process for an online panel contribute to nonresponse and selection bias?
(2) How do selection bias and attrition evolve over time in a high-frequency online panel survey?
(3) What are the survey costs associated with recruitment and panel stability?
Using detailed individual-level data from complete employment biographies allows for precise analysis of how biases and nonresponse emerge throughout the recruitment process. Additionally, it facilitates tracking the evolution of selection bias and panel attrition across socioeconomic and demographic groups over successive survey waves. Finally, I calculate average costs for recruiting one-time respondents and one-year panelists, providing valuable insights into cost-efficiency by demographic and socioeconomic profiles.
Dr Liliya Leopold (University of Amsterdam) - Presenting Author
Mrs Noble Nolen (University of Amsterdam)
Mr Diego Strassmann Rocha (University of Bremen)
Dr Thomas Leopold (University of Cologne)
Dr Brian O'Shea (University of Nottingham)
This study assesses the accuracy of findings on self-rated health (SRH) and its demographic variations across gender, education, ethnicity, race, and age in non-probability surveys, evaluating their potential to advance research on health disparities.
We analyzed three non-probability surveys with extensive health measures: the Research and Development Survey 8 (RANDS-8), the Quota-based Population Health Survey (QPHS), and the Project Implicit – Health Survey (PI-H). These surveys were benchmarked against the U.S. Census’s Current Population Survey (CPS) and compared with four probability-based surveys: the National Health Interview Survey (NHIS), National Health and Nutrition Examination Survey (NHANES), General Social Survey (GSS), and the Understanding America Study (UAS). Data from 141,008 respondents aged 18 to 65, collected between 2022 and 2024, were included.
While non-probability surveys generally exhibited lower accuracy than probability-based surveys, univariate SRH distributions in RANDS-8 and QPHS, along with certain multivariate estimates—such as age-SRH and education-SRH associations in RANDS-8 and QPHS, and Black race-SRH associations in PI-H—showed strong alignment with the benchmark. Conversely, gender-SRH associations in all non-probability surveys substantially overstated effect sizes compared to benchmarks, while ethno-racial disparities in SRH in RANDS-8 and QPHS diverged strongly, even reversing direction.
These findings underscore both the promise and the challenges of using non-probability surveys to study SRH and its demographic variations. They highlight the need for rigorous validation against reliable benchmarks to ensure accurate insights into health disparities.
Ms Jessica Donzowa (Max Planck Institute for Demographic Research/ Bielefeld University) - Presenting Author
Professor Simon Kühne (Bielefeld University)
Ms Zaza Zindel (Bielefeld University)
Researchers are increasingly using social media platforms for survey recruitment. However, empirical evidence remains sparse on how the content and design characteristics of advertisements used for recruitment affect response quality in surveys. Building on leverage-salience and self-determination theory, we assess the effects of advertisement design on response quality. We argue that different advertisement designs may resonate with specific social groups who vary in their commitment to the survey, resulting in differences in the observed response quality. We use data from a survey experiment conducted via ads placed on Facebook in Germany and the United States in June 2023. A commercial access panel company was contracted to include identical survey questions to allow for comparison with the Facebook recruited survey data. The survey, focusing on attitudes toward climate change and immigration, featured images with varying thematic associations with the topics (strong, loose, neutral). The Facebook sample consisted of 4,170 respondents in Germany and 5,469 respondents in the United States. We compare several data quality indicators, including break-off rate, completion time, non-differentiation, item non-response, passing an attention check question, and follow-up availability, across different advertisement features. Regression analyses indicate differences in response quality across advertisement designs, with a strong thematic design generally being associated with poorer response quality. Strongly themed ad designs are generally associated with higher attrition, non-differentiation, and item non-response, and with a lower probability of passing an attention check and providing an e-mail address for future survey inquiries. Our study advances the literature by highlighting the substantial impact of advertisement design on survey data quality, and emphasizing the importance of tailored decision-making in recruitment design for social media-based survey research.