ESRA 2025 Preliminary Program
All time references are in CEST
Get in touch–stay in touch: Efficient and suitable recruitment and survey strategies 2 |
| Session Organisers |
Dr Roman Auriga (Leibniz Institute for Educational Trajectories)
Mr André Müller-Kuller (Leibniz Institute for Educational Trajectories)
Ms Anna Passmann (Leibniz Institute for Educational Trajectories)
|
| Time | Wednesday 16 July, 13:30 - 15:00 |
| Room |
Ruppert 042 |
This year's session will focus on appropriate and efficient recruitment and fieldwork strategies in current population-representative surveys. In both individual and institutional survey settings, recruitment of survey participants has been under "pressure to change" for several years. On the one hand, we face declining response rates in surveys. On the other hand, recruiting participants seems to have become more and more expensive over the years due to inflation, salary developments and change in employment relationships regarding the survey stuff or more complex survey requirements and designs. Social changes must also be taken into account: Due to the increasing factual and perceived heterogeneity of societies, expectations regarding the characteristics of the target persons to be recruited and the ways how to address them appropriately are also becoming increasingly complex. Suitable recruitment and survey designs must therefore be found.
In this session, we want to discuss innovations in survey designs and methods under the premise of optimizing and balancing survey success and costs. In particular, tailored recruitment strategies in terms of mixed approaches (e.g. sequential designs and/or pre-recruitment strategies), incentive strategies (e.g. pre-incentives, promised or perceived benefits of participation), contact strategies (e.g. adaptations in the language used, layout of materials, new media), but also field work strategies and interviewer tasks should/could be at focus.
We welcome contributions with insights into the recruitment of large-scale studies of the last five years. “Quantitative” findings as well as well documented “qualitative” experiences are both welcome.
Keywords: recruitment, efficiency, costs, innovation, response rates, tailored designs
Papers
Are Phone Call Reminders Worth the Cost in Mixed-Mode Surveys?
Mrs Noémie Soullier (Santé publique France) - Presenting Author
Mrs Leïla Saboni (Santé publique France)
Mrs Maria El Haddad (Santé publique France)
Mrs Axelle Quiviger (Santé publique France)
Mr Jean-Baptiste Richard (Santé publique France)
To face the challenge of declining response rates, survey designers must build cost-effective protocols. One way to do so is by mixing modes to collect data and contact participants. In a mixed-mode survey, we evaluated the effectiveness of a phone call to encourage online participation. We expected this phone reminder to accelerate and improve participation, especially online. However, proving its effectiveness is crucial given its cost (twice that of a mail and ten times that of an email).
The French Health Barometer is a cross-sectional repeated survey conducted by the French Public Health Agency, interviewing the population living in France about their health-related opinions, behaviours and knowledge. The sample is randomly drawn from the demographic file of dwellings and individuals. In the 2024 survey, we randomly assigned participants to two groups. Both groups followed a sequential mixed-mode protocol. It began with a web questionnaire for 5 weeks, followed by 10 weeks offering a choice between the web questionnaire and a phone interview. Invitations and reminders were sent by mail and email. The treatment group (54,000) received an additional phone reminder during the web-only phase: two weeks after the survey began, a phone call encouraged participants to respond online. The control group (7,000) did not receive this phone reminder.
In this presentation, we propose a cost-effectiveness analysis of a phone reminder in the web-only phase of a mixed-mode protocol. We describe its impact on response rates and participants' preferred modes of completion, with distinctions by region (mainland vs overseas) and respondents’ socio-economic characteristics. We also examine how the supplementary reminder influenced the socio-demographic composition of the respondents’ sample. Finally, we compare the costs of the two protocols and suggest recommendations for using phone reminders in future surveys.
Different Modes, Different Outcomes – Recruiting Respondents for the Panel Study on the Back of ESS
Mrs Jaroslava Pospíšilová (Institute of Sociology, Czech Academy of Sciences) - Presenting Author
This paper focuses on the effectiveness of different data collection modes in recruiting respondents for a panel study by comparing the last two rounds of the European Social Survey (ESS) in the Czech Republic. In the last two rounds of the ESS, data collection included recruiting respondents for the Opinion Survey (CRONOS) panel. While data for ESS R10 was collected using the face-to-face (F2F) method, challenges stemming from the disintegration of interviewer networks and increased public distrust following the COVID-19 pandemic prompted in the Czech Republic a shift to a self-completion method for ESS R11. Initially, a sequential approach was adopted, but during data collection, it was replaced by a concurrent approach, which proved more effective in reaching populations with limited internet access. Although F2F recruitment was expected to achieve higher response rates, the actual number of participating respondents in ESS R10 was rather disappointing. About 30% of invited respondents agreed to participate, but only 25% of them completed the initial online questionnaire. Conversely, recruitment for ESS R11 using the self-completion method yielded significantly better outcomes. Over 50% of respondents accepted the invitation to the follow-up study, and more than 70% of them completed the first survey. Moreover, data indicate that offering respondents the option to participate via paper questionnaires is the least advantageous strategy, as it is associated with higher rates of invalid contacts and increased attrition from the study.
Individualized Results for Schools as Effective Incentive? – Findings from an Evaluation in NEPS-Schools
Mrs Loreen Beier (Leibniz Institute for Educational Trajectories) - Presenting Author
Mrs Anna Passmann (Leibniz Institute for Educational Trajectories)
Individualized results for schools as an incentive have been implemented in the National Educational Panel Study (NEPS) with the beginning of the Starting Cohort 8 (SC8) in 2022. All schools invited to participate in NEPS were offered these feedback materials with individualized results and comparisons to other participating schools of the same school type and region. The feedback is presented in yearly created posters und brochures with changing main topics.
Most research about individualized results for schools focusses on cross-sectional school performance studies or evaluations (e. g. PISA, MARKUS, IGLU) which is not our goal. Instead, we offer an incentive for long-term participation. But so far, little is known about how incentives of this kind affect long-term willingness to participate (cf. Batinic & Moser, 2005). However, some studies note the potential of annual reports for schools to be an effective incentive (cf. e. g. Haelermans et al., 2020; Schneewind, 2007).
In group discussions about materials for participants in advance of SC8 teachers and principals showed high interest in individualized results for schools. In the current third wave of SC8 we asked representatives from the participating NEPS schools how they perceive this kind of incentive. Results from this evaluation will be presented in the session.
Batinic, B., & Moser, K. (2005). Determinanten der Rücklaufquote in Online-Panels. Zeitschrift für Medienpsychologie, 17(2), 64–74.
Haelermans, C., Huijgen, T., Jacobs, M., Levels, M., van der Velden, R., van Vugt, L., & van Wetten, S. (2020). Using Data to Advance Educational Research, Policy, and Practice: Design, Content, and Research Potential of the Netherlands Cohort Study on Education. European Sociological Review, 36(4), 643–662.
Schneewind, J. (2007). Wie Lehrkräfte mit Ergebnisrückmeldungen aus Schulleistungsstudien umgehen.: Ergebnisse aus Befragungen von Berliner Grundschullehrerinnen.
The key role of panel managers in enhancing panelist engagement
Miss Sarah Blain (Centre de Données Socio-politiques - Sciences Po Paris)
Miss Justine Lévy (Centre de Données Socio-politiques - Sciences Po Paris) - Presenting Author
In the chain of quantitative data production in social sciences, panel management remains a little-known yet essential link. Panel managers play a crucial role in maintaining high survey participation rates and limiting attrition through various strategies: personalized reminders, dedicated contact with panellists, incentive campaigns, and newsletters. Acting as intermediaries between panellists and data production or research teams, they ensure the continuity of interactions while safeguarding participants' anonymity.
This study is based on a cross-analysis of two panels (ELIPSS and CRONOS) managed by the Centre de données socio-politiques (Sciences Po Paris - CNRS), which feature contrasting approaches to incentives, reminders, and panel engagement. It examines the impact of these mechanisms on the nature and frequency of interactions between managers and panellists, particularly regarding practical questions, complaints, and participant satisfaction. These findings shed light on how incentives influence respondent loyalty and the quality of relationships established with them.
The analysis also explores feedback provided by panellists on the questionnaires, especially through open-ended questions at the end of surveys. These contributions, more frequently shared by certain respondent profiles (with consideration of their socio-demographic characteristics), raise questions about their role: are they merely a limited outlet or a valuable source for improving survey tools? These results provide an opportunity to reflect on how panel managers, as direct intermediaries with panellists, can contribute to improving methodological practices and protocols, whether longitudinal or cross-sectional.
