All time references are in CEST
Get in touch–stay in touch: Efficient and suitable recruitment and survey strategies |
|
| Session Organisers | Dr Roman Auriga (Leibniz Institute for Educational Trajectories) Mr André Müller-Kuller (Leibniz Institute for Educational Trajectories) Ms Anna Passmann (Leibniz Institute for Educational Trajectories) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
This year's session will focus on appropriate and efficient recruitment and fieldwork strategies in current population-representative surveys. In both individual and institutional survey settings, recruitment of survey participants has been under "pressure to change" for several years. On the one hand, we face declining response rates in surveys. On the other hand, recruiting participants seems to have become more and more expensive over the years due to inflation, salary developments and change in employment relationships regarding the survey stuff or more complex survey requirements and designs. Social changes must also be taken into account: Due to the increasing factual and perceived heterogeneity of societies, expectations regarding the characteristics of the target persons to be recruited and the ways how to address them appropriately are also becoming increasingly complex. Suitable recruitment and survey designs must therefore be found.
In this session, we want to discuss innovations in survey designs and methods under the premise of optimizing and balancing survey success and costs. In particular, tailored recruitment strategies in terms of mixed approaches (e.g. sequential designs and/or pre-recruitment strategies), incentive strategies (e.g. pre-incentives, promised or perceived benefits of participation), contact strategies (e.g. adaptations in the language used, layout of materials, new media), but also field work strategies and interviewer tasks should/could be at focus.
We welcome contributions with insights into the recruitment of large-scale studies of the last five years. “Quantitative” findings as well as well documented “qualitative” experiences are both welcome.
Keywords: recruitment, efficiency, costs, innovation, response rates, tailored designs
Mrs Noémie Soullier (Santé publique France) - Presenting Author
Mrs Leïla Saboni (Santé publique France)
Mrs Maria El Haddad (Santé publique France)
Mrs Axelle Quiviger (Santé publique France)
Mr Jean-Baptiste Richard (Santé publique France)
To face the challenge of declining response rates, survey designers must build cost-effective protocols. One way to do so is by mixing modes to collect data and contact participants. In a mixed-mode survey, we evaluated the effectiveness of a phone call to encourage online participation. We expected this phone reminder to accelerate and improve participation, especially online. However, proving its effectiveness is crucial given its cost (twice that of a mail and ten times that of an email).
The French Health Barometer is a cross-sectional repeated survey conducted by the French Public Health Agency, interviewing the population living in France about their health-related opinions, behaviours and knowledge. The sample is randomly drawn from the demographic file of dwellings and individuals. In the 2024 survey, we randomly assigned participants to two groups. Both groups followed a sequential mixed-mode protocol. It began with a web questionnaire for 5 weeks, followed by 10 weeks offering a choice between the web questionnaire and a phone interview. Invitations and reminders were sent by mail and email. The treatment group (54,000) received an additional phone reminder during the web-only phase: two weeks after the survey began, a phone call encouraged participants to respond online. The control group (7,000) did not receive this phone reminder.
In this presentation, we propose a cost-effectiveness analysis of a phone reminder in the web-only phase of a mixed-mode protocol. We describe its impact on response rates and participants' preferred modes of completion, with distinctions by region (mainland vs overseas) and respondents’ socio-economic characteristics. We also examine how the supplementary reminder influenced the socio-demographic composition of the respondents’ sample. Finally, we compare the costs of the two protocols and suggest recommendations for using phone reminders in future surveys.
Dr Jean Philippe Décieux (University of Bonn & Federal Institute for Population Research (BiB)) - Presenting Author
Dr Andreas Ette (Federal Institute for Population Research (BiB))
his study examined whether survey participation among (temporary) dropouts from previous waves could be influenced by targeted incentives. For this purpose, two split-ballot experiments were conducted within this group of (temporary) dropouts: In wave 4, we compared the effects of switching from a conditional 5-Euro post-paid incentive to a 10-Euro incentive versus consistently offering a 5-Euro incentive. In wave 5, the initial 5-Euro group received 10 Euros, while the 10-Euro group from wave 4 was split into two subgroups—one continuing to receive 10 Euros and the other reverting to 5 Euros. The experiments were carried out within the BiB/FReDA survey “Refugees from Ukraine,” a probability-based register sample. The key performance indicators of our analysis will be wave-specific and cumulative response rates. Additionally, we reflects aspects of data quality, demographic composition, and survey costs.
Preliminary results suggest that targeted incentive increases positively impacted participation among the (temporary) dropout group. As in wave 4, participation was higher when a 10-Euro incentive was offered. However, our second experiment complemented that the increase in participation was primarily driven by the higher incentive: response rates were highest among those who received 10 Euros in both waves, followed by the group that switched from 5 Euros in wave 4 to 10 Euros in wave 5. Both groups outperformed the group that switched from 10 Euros in wave 4 to 5 Euros in wave 5.
These initial findings suggest that higher incentives can effectively re-engage (temporary) dropouts. However, to maintain this positive effect and ensure panel stability, consistently offering higher incentive values may be necessary.
Ms Theresa Müller (infas Institut für angewandte Sozialwissenschaft GmbH (Institute for Applied Social Sciences)) - Presenting Author
Mrs Tabea Feseker (infas Institut für angewandte Sozialwissenschaft GmbH (Institute for Applied Social Sciences))
Mr Michael Ruland (infas Institut für angewandte Sozialwissenschaft GmbH (Institute for Applied Social Sciences))
Face-to-face surveys are considered the "gold standard" in social research but are also associated with high costs, particularly in initial survey waves due to the high contact effort required by interviewers. This paper examines the effectiveness of prerecruitment strategies in three large-scale surveys in Germany: the 2023 refreshment sample of the IAB-BAMF-SOEP survey of refugees, a NEPS linking study, and the first wave of the DaLi study. The goal was to gain additional contact information and minimize selection bias caused by non-contact.
Each survey used different strategies to enhance response rates in the prerecruitment phase, such as prepaid incentives and personalized reminder letters at different stages of the prerecruitment phase. This paper analyzes how effective each of these strategies was and tries to give a best practice advise for future surveys.
Since prerecruitment should not increase but decrease the selection bias in face-to-face surveys, this paper also examines the effects of the prerecruitment strategies on the selection bias. For this, the socio-demographic characteristics are compared between prerecruited and non-prerecruited target persons. The aim of this analysis is to establish whether additional actions need to be taken in the face-to-face field to contact and interview non-prerecruited target persons.
Mr André Müller-Kuller (Leibniz Institute for Educational Trajectories) - Presenting Author
Ms Viktoria Zorn (Leibniz Institute for Educational Trajectories)
The available budget and resources significantly influence how and what data can be collected, as well as its quality. This fundamental judgement, impressively illustrated by Groves in 1989, is still valid and continues to gain importance. Over the past decades, costs and efforts associated with surveys have steadily increased, suggesting that the cost of data collection will continue to rise. Consequently, the question of possibilities and potential for controlling these costs arises. For researchers, especially those who design and conduct surveys, understanding how various cost drivers affect both costs and quality and/or errors at all stages of the process is crucial. With regard to efficient survey designing, the key questions remain or have become even more urgent: How do costs vary across different types and designs of surveys? How do different design features (components) impact costs? The decomposition of survey costs is challenging, as it depends on the availability of information (transparency), the complexity of the surveys themselves, and the comparability of costs. Early survey cost approaches focussed on individual survey and/or cost components with limited scope and practical restrictions in terms of implementation options. Although the development and variation of survey costs have increasingly attracted attention in recent years, data and research that explicitly address survey costs remain scarce. This applies in particular to surveys that are commissioned rather than carried out by the researchers themselves and especially to panel studies. In this paper we examine survey costs across different cohorts of a panel study (NEPS). Design and cost information has been collected over more than a decade at various aggregation levels. Drawing on the typology by Olson et al. (2021), we examine costs components of commissioned interviewer-administered surveys (CATI/CAPI) and analyze cost and design factors with regard to possibilities and potential for increasing efficiency.
Dr Jonas Klingwort (Statistics Netherlands (CBS) - Department of Research & Development) - Presenting Author
Mr Jeldrik Bakker (Statistics Netherlands (CBS) - Department of Research & Development)
Dr Vera Toepoel (Statistics Netherlands (CBS) - Department of Research & Development)
Sample surveys are important for official statistics as they provide a structured method of collecting data on various topics, enabling governments, organizations, and researchers to make informed decisions and monitor societal changes. However, response rates are declining, and understanding which factors influence response rates is crucial for improving survey participation and reducing potential nonresponse bias. This study investigates which features significantly affect response rates and how surveys can be optimized to improve participation.
We conducted a multilevel meta-analysis using Statistics Netherlands’ data from 38-person population surveys with about 1,200 independent samples. The surveys were fielded over seven years (2018–2024) with a total sample size of over 7 million people. These surveys range from one-time to recurring studies with frequencies from weekly to biennially. We used about 60 features such as respondent factors (e.g., age, gender, nationality, device use), survey design factors (e.g., year, month, mode, incentive, fieldwork period, number of contacts, topic), and questionnaire factors (e.g., duration, number of blocks, pages & questions, number of question per question type, number of introduction texts) that potentially have an effect on the response rates.
Preliminary findings suggest that the data collection mode, type of incentives, survey topic, the number of and types of questions, the device used by the respondent, and age and gender have significant effects on the response rate. Interestingly, the length of the fieldwork, the number of reminders, and the periodicity of the survey show non-significant effects.
This study offers comprehensive insights into factors affecting response rates in official statistics surveys. It highlights how effective data collection strategies should be designed and identifies survey designs to avoid. When designing surveys, the study is a practical toolbox for national statistical agencies, survey agencies, and survey researchers.
Dr Claudia Schmiedeberg (LMU Munich) - Presenting Author
Dr Christiane Bozoyan (LMU Munich)
Dr Jette Schröder (GESIS)
As high response rates are crucial for surveys to produce high quality data, much effort is put into maximizing response rates. In doing so, trust is a critical factor in gaining the cooperation of respondents (Dillman et al., 2014). Media coverage is thought to be an important way for a survey to
demonstrate seriousness and build trust. Accordingly, survey methods textbooks suggest accompanying fieldwork with press releases (e.g., Reuband 2019), and many surveys have followed this advice (e.g., German Sex Survey GeSiD: Matthiesen et al. 2021; German Health Update GEDA:
Lange et al., 2017).
However, to our knowledge, this presumed effect of media coverage on survey participation has not been analyzed yet. We use the delayed media coverage of the recruitment survey of the German Longitudinal Environmental Survey (GLEN), a large-scale panel survey that began in 2024 with a
recruitment survey of a gross sample of 80,000 individuals residing in Germany, to examine whether press articles published approximately one week after the start of fieldwork altered response patterns.
The GLEN project did not send out press releases prior to the fieldwork, but the largest German tabloid newspaper, ‘Bild’, became aware of the mail invitations, which included a €5 prepaid incentive. The newspaper contacted the project team and reported on the project in a rather neutral
tone on its online news channel. Subsequently, several other local newspapers also reported on the study.
We analyze the development of participation over the fielding period as well as the distribution of responses across regions and educational groups as we expect that these reports will increase participation especially (1) among respondents with lower education, who are the main target group of
"Bild", and (2) in the regions where the local newspapers are popular.
Professor Lauri Rapeli (Åbo Akademi University)
Mr Kim Backström (Åbo Akademi University) - Presenting Author
Mr Marek Berg (Åbo Akademi University)
Getting people to participate in surveys is a persistent problem for all survey researchers. Unlike ordinary citizens, elected politicians are seldom the target population for survey recruitment efforts. In many countries, politicians often are not prepared to volunteer as participants in surveys, let alone panels, which leads to a lack of studies regarding their willingness to collaborate in surveys.
This study reports the findings from a rare survey recruitment experiment in which 7,397 local-level politicians from Finland were randomly assigned to groups that received by e-mail either 1) an altruistic or 2) an egoistic appeal or 3) an appeal referring to benefits for science when they were invited to participate in the Finnish Politician Panel.
While there are some statistically significant differences across appeals among sub-groups, overall, we find no differences in recruitment rates. This suggests that, at least in politician samples, the altruism-egoism framework is largely irrelevant and that for maximizing participation rates among politicians, survey researchers should look into alternative methods. However, politicians who are older, live in urban municipalities, and represent the political left are more likely than their counterparts to volunteer as panel participants.
By examining recruitment propensities and the effects of appeals, this study sheds light on representation in an understudied population. The findings provide practical insights for designing future recruitment efforts and improving response rates among politicians.
Mrs Jaroslava Pospíšilová (Institute of Sociology, Czech Academy of Sciences) - Presenting Author
This paper focuses on the effectiveness of different data collection modes in recruiting respondents for a panel study by comparing the last two rounds of the European Social Survey (ESS) in the Czech Republic. In the last two rounds of the ESS, data collection included recruiting respondents for the Opinion Survey (CRONOS) panel. While data for ESS R10 was collected using the face-to-face (F2F) method, challenges stemming from the disintegration of interviewer networks and increased public distrust following the COVID-19 pandemic prompted in the Czech Republic a shift to a self-completion method for ESS R11. Initially, a sequential approach was adopted, but during data collection, it was replaced by a concurrent approach, which proved more effective in reaching populations with limited internet access. Although F2F recruitment was expected to achieve higher response rates, the actual number of participating respondents in ESS R10 was rather disappointing. About 30% of invited respondents agreed to participate, but only 25% of them completed the initial online questionnaire. Conversely, recruitment for ESS R11 using the self-completion method yielded significantly better outcomes. Over 50% of respondents accepted the invitation to the follow-up study, and more than 70% of them completed the first survey. Moreover, data indicate that offering respondents the option to participate via paper questionnaires is the least advantageous strategy, as it is associated with higher rates of invalid contacts and increased attrition from the study.
Mrs Loreen Beier (Leibniz Institute for Educational Trajectories) - Presenting Author
Mrs Anna Passmann (Leibniz Institute for Educational Trajectories)
Individualized results for schools as an incentive have been implemented in the National Educational Panel Study (NEPS) with the beginning of the Starting Cohort 8 (SC8) in 2022. All schools invited to participate in NEPS were offered these feedback materials with individualized results and comparisons to other participating schools of the same school type and region. The feedback is presented in yearly created posters und brochures with changing main topics.
Most research about individualized results for schools focusses on cross-sectional school performance studies or evaluations (e. g. PISA, MARKUS, IGLU) which is not our goal. Instead, we offer an incentive for long-term participation. But so far, little is known about how incentives of this kind affect long-term willingness to participate (cf. Batinic & Moser, 2005). However, some studies note the potential of annual reports for schools to be an effective incentive (cf. e. g. Haelermans et al., 2020; Schneewind, 2007).
In group discussions about materials for participants in advance of SC8 teachers and principals showed high interest in individualized results for schools. In the current third wave of SC8 we asked representatives from the participating NEPS schools how they perceive this kind of incentive. Results from this evaluation will be presented in the session.
Batinic, B., & Moser, K. (2005). Determinanten der Rücklaufquote in Online-Panels. Zeitschrift für Medienpsychologie, 17(2), 64–74.
Haelermans, C., Huijgen, T., Jacobs, M., Levels, M., van der Velden, R., van Vugt, L., & van Wetten, S. (2020). Using Data to Advance Educational Research, Policy, and Practice: Design, Content, and Research Potential of the Netherlands Cohort Study on Education. European Sociological Review, 36(4), 643–662.
Schneewind, J. (2007). Wie Lehrkräfte mit Ergebnisrückmeldungen aus Schulleistungsstudien umgehen.: Ergebnisse aus Befragungen von Berliner Grundschullehrerinnen.
Miss Sarah Blain (Centre de Données Socio-politiques - Sciences Po Paris)
Miss Justine Lévy (Centre de Données Socio-politiques - Sciences Po Paris) - Presenting Author
In the chain of quantitative data production in social sciences, panel management remains a little-known yet essential link. Panel managers play a crucial role in maintaining high survey participation rates and limiting attrition through various strategies: personalized reminders, dedicated contact with panellists, incentive campaigns, and newsletters. Acting as intermediaries between panellists and data production or research teams, they ensure the continuity of interactions while safeguarding participants' anonymity.
This study is based on a cross-analysis of two panels (ELIPSS and CRONOS) managed by the Centre de données socio-politiques (Sciences Po Paris - CNRS), which feature contrasting approaches to incentives, reminders, and panel engagement. It examines the impact of these mechanisms on the nature and frequency of interactions between managers and panellists, particularly regarding practical questions, complaints, and participant satisfaction. These findings shed light on how incentives influence respondent loyalty and the quality of relationships established with them.
The analysis also explores feedback provided by panellists on the questionnaires, especially through open-ended questions at the end of surveys. These contributions, more frequently shared by certain respondent profiles (with consideration of their socio-demographic characteristics), raise questions about their role: are they merely a limited outlet or a valuable source for improving survey tools? These results provide an opportunity to reflect on how panel managers, as direct intermediaries with panellists, can contribute to improving methodological practices and protocols, whether longitudinal or cross-sectional.
Ms Birgit Jesske (infas Institute for Applied Social Sciences)
Mr Michael Ruland (infas Institute for Applied Social Sciences) - Presenting Author
Interviewers are an important team player during fieldwork. They have to deal with the complex survey designs of different studies and fulfill the specific requirements in each study. And last but not least, they are confronted with the low willingness and poor reachability of respondents.
Are all interviewers affected in the same way by the changed survey conditions of recent years? Even if numerous deployment strategies are used by survey institutes or special studies, it is generally known that different interviewers have different response rates. In many panel surveys, interviewer continuity is considered in order to avoid panel attrition. Interviewers are deployed close to their place of residence, not least for cost reasons and may be subject to access restrictions that vary from region to region. However, experienced interviewers are sometimes more successful because they have a broader range of contact strategies to convince reluctant respondents.
With the interviewer administration file at infas, we have a database with which we can analyze the performance of over 3,000 interviewers over the last 10 years, both in the context of panel surveys and large-scale cross-sectional surveys or ad hoc studies. In our multivariate analysis model, we can control the development of response rates over time, both according to the design of the studies and the characteristics of the interviewers. In addition, contact log files from the panel study “Labor Market and Social Security” (PASS) provide further insights into the contact strategies and the contact effort that interviewers had to make over the last 10 years to both re-contact panel participants and recruit new participants.
With our paper we would like to provide further insights into the tasks and commitment of interviewers in fieldwork and contribute additional aspects to the discussion of suitable recruitment and survey strategies.