All time references are in CEST
Recruitment methods for surveys without field interviewers |
|
| Session Organisers | Dr Cristian Domarchi (University of Southampton) Dr Olga Maslovskaya (University of Southampton) Mr Andrew Phelps (Office for National Statistics) Dr Mariel Leonard (DIW-Berlin) Dr Carina Cornesse (DIW-Berlin) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
In many countries, data collection organisations are undergoing a paradigm shift, with social surveys, including censuses, experiencing major transformations in their design and implementation. Prior to the COVID-19 pandemic, some data collection agencies started moving towards online data collection, as response rates in social surveys had been falling and fieldwork costs were increasing. The pandemic provided an opportunity to move surveys to self-administered modes with unprecedented speed and expedited existing transformation plans. However, some social surveys did not make this transition and several returned to face-to-face interviewing as soon as it was feasible to do so.
In this session, we would like to investigate the main barriers to effective recruitment in self-completion surveys in both cross-sectional and longitudinal contexts. Topics may include but are not restricted to the following areas:
• Recruitment strategies and practices.
• Knock-to-nudge and other innovations in recruitment methods.
• Targeted survey procedures for recruitment in self-completion surveys.
• Inclusivity and accessibility practices to engage population subgroups.
We encourage papers from researchers with a variety of backgrounds and across different sectors, including academia, national statistics, and research agencies. We particularly welcome contributions using experimental designs, and/or other methods that can inform future strategies for recruitment in self-completion surveys.
The session is proposed by Research Strand 4 of the Survey Futures project, “Surveys without Field Interviewers”. Survey Futures is a UKRI-ESRC funded research programme focused on ensuring large-scale social surveys in the UK can innovate and adapt in a changing environment. Survey Futures is a multi-institution collaboration between universities and survey practice organisations.
Keywords: self-administered surveys, online surveys, survey recruitment, sample representativeness, response rates
Mr Tim Hanson (European Social Survey HQ (City St Georges, University of London)) - Presenting Author
Mr Nathan Reece (European Social Survey HQ (City St Georges, University of London))
Ms Clare Kavanagh (Ipsos B&A)
Ms Katie Kirkwood (Ipsos B&A)
Mr Ciaran McGuirl (Ipsos B&A)
Mr Luke Reaper (Ipsos B&A)
The European Social Survey (ESS) is preparing to transition to self-completion (web and paper) data collection, having previously used face-to-face interviewing. To prepare for this switch, the self-completion approach has been tested in several countries. This included a pilot in Ireland – a country with little previous experience of equivalent self-completion approaches – in the summer of 2024.
Under the new approach, most ESS countries will send letters in the post to recruit people to complete the survey via self-completion. There are a small number of countries where postal services are unreliable, or sample frames don’t enable delivery of letters to specific addresses. In these cases, fieldworkers will be used to hand-deliver letters to sampled addresses (‘knock-to-nudge’).
Ireland presents a rare situation where postal recruitment is possible for most addresses but not for others due to the presence of non-unique addresses. This meant taking a hybrid approach whereby two-thirds of cases were recruited by post and the other third by fieldworker visits.
The hybrid approach allows us to assess the overall experience of piloting the self-completion approach in Ireland and compare the two recruitment methods used. We found that both approaches delivered similarly good response rates, with over 40% achieved for each method.
Our paper will: provide an overview of the designs followed for both approaches; assess outcomes (response rates, sample compositions) overall and between approaches; provide feedback based on experiences of fieldworkers delivering the knock-to-nudge approach; present information on the timeliness and cost effectiveness of each approach; and share recommendations regarding future implementation.
It is expected that results will be of significant interest to those implementing similar self-completion approaches, especially in cases where recruitment cannot only rely on postal mailings.
Mr Georg-Christoph Haas (Institute for Employment Research) - Presenting Author
Mr Benjamin Baisch (Institute for Employment Research)
Mr Mark Trappmann (Institute for Employment Research)
Mr Jonas Weik (Institute for Employment Research)
In online panels, emails are a crucial element for recruiting respondents. Email invitations may substantially affect panelists’ perception of the study's relevance, potentially influencing both response rates and sample composition. Previous research has examined the use of targeted appeals, where the wording in the invitation letter varies among pre-identified subgroups. In our study, we divide panelists into subgroups based on their self-stated motivations to participate. We then use these self-stated motivations to craft an appealing email invitation to invite panelists to a subsequent wave. Based on the Leverage-Saliency Theory, emphasizing the self-stated motivation in email invitations (Saliency) should have a positive effect on the panelists' response propensity, enhancing cooperation as well as reducing attrition within the panel. Our design enables us to answer the question: Do targeted invitations based on panelists self-stated motivations from a previous wave increase response rates in a subsequent wave? We implemented a survey experiment in a German Online Probability Panel: IAB-OPAL. In wave 3, we asked 10,246 panelists to state their main motivation for participation, choosing among seven different motivations: topic, incentive, giving opinion, informing politics, curiosity, helping science, feeling obligated. In wave 4, we randomly assigned panelist either to the standard invitation or to an invitation that aligns with one of the self-stated motivations. Our treatment included a different subject line as well as a motivational email text. Results show that our treatment was not able to improve cooperation or reduce attrition within the panel. On the contrary, for the motivations “giving opinion” and “informing politics”, our results show that aligning the wording of the invitation email with panelists self-stated motivations from the previous wave reduces response rates compared to the standard invitation email.
Dr Lisa Schmid (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Professor Tobias Gummer (GESIS - Leibniz Institute for the Social Sciences)
The aim of our study is to identify relationships between respondent and survey characteristics with panel consent to improve effectiveness of panel recruitment processes. Obtaining consent to participate in subsequent panel waves is a crucial step when recruiting panelists. Survey projects devote considerable resources establishing contact with potential respondents and ensure their initial participation. If survey researchers cannot obtain panel consent in the initial interview, these resources and efforts are in vain. To address the consent challenge for panel surveys, we relied on data from the recruitment wave of the FReDA – The German Family Demography Panel Study, which is a large-scale mixed-mode probability-based panel in Germany. We tested the impact of various respondent and survey-related characteristics on panel consent. Based on our results we showcase which respondent groups are at risk of not providing consent and which survey characteristics could be altered to improve panel consent rates.
Our findings show that the step of obtaining respondents consent to participate in subsequent panel waves during the initial recruitment survey is a crucial challenge to panel surveys. After the completion of the recruitment wave, there is a considerable number of respondents willing to participate in a survey who reject the request of being interviewed again in subsequent waves and thus, lost potential for the panel survey. Furthermore, our results show that the consent process is selective. Respondents differ based on their characteristics whether they offer panel consent or not. Our study pinpoints risk groups most likely not to offer panel consent. In addition, we show that the design and content of the questionnaire is a viable way of improving panel consent. Using a more interesting and shorter questionnaire that is not perceived as too personal is likely to enhance participation rates.
Dr Christiane Bozoyan (LMU Munich) - Presenting Author
Dr Claudia Schmiedeberg (LMU Munich)
Dr Jette Schröder (GESIS)
This study investigates whether the inclusion of a mascot in a reminder letter increases survey response rates. The experiment is conducted in a survey of the German Longitudinal Environmental Study (GLEN) in January 2025. Participants are randomly assigned to two groups after the initial survey invitation: the treatment group (N=1000) receives a (postal) reminder letter with a prominently displayed mascot, while the rest of the sample receives the, otherwise identical, reminder without the mascot. The mascot is designed to evoke positive emotions and foster emotional connection with the study, potentially increasing participation through reciprocity.
The effectiveness of the mascot reminder is measured by comparing response rates between the two groups. An increased response rate in the mascot group would suggest that a mascot can indeed influence survey participation. Moreover, we investigate whether the mascot increases participation differentially among subgroups, e.g. different in sociodemographic characteristics and attitudes. The findings provide insights for future survey designs as they demonstrate the potential of incorporating elements evoking sympathy to increase participation. If successful, this approach this approach could improve data quality and reduce costs.
Dr Olga Maslovskaya (University of Southampton) - Presenting Author
Dr Cristian Domarchi (University of Southampton)
Professor Peter Smith (University of Southampton)
The knock-to-nudge is an innovative method of household contact, first introduced during the COVID-19 pandemic when face-to-face interviewing was not possible. In this approach, interviewers visit households and encourage sampled units to participate in a survey through a remote survey mode (either web or telephone) at a later date. Interviewers may also collect contact information, such as telephone numbers or email addressed, or conduct within-household selection of individuals on the doorstep. This approach continued to be used post-pandemic, but there remains a knowledge gap regarding its advantages and disadvantages. It is still unclear whether knock-to-nudge approach leads to improvements in sample composition and data quality.
We analyse data from three UK surveys: the National Survey for Wales (NSW), the Transformed Labour Force Survey (TLFS), and the National Readership Survey (PAMCo), each of which employed different versions of the knock-to-nudge approach. Our goal is to assess whether this method improves sample composition across these three surveys. We begin with descriptive analysis and then apply logistic regression models to investigate the composition of subsamples that received different recruitment treatments. Finally, we compare the sample composition at various stages of the recruitment process.
Our preliminary findings suggest that the knock-to-nudge approach is effective in improving sample composition, while also in reducing costs compared to traditional face-to-face interviews.
This study contributes to the under-researched area of knock-to-nudge methods. The results indicate that, when carefully designed and implemented, this approach can enhance recruitment efforts and improve sample composition of the resulting samples in surveys.
Dr Cristian Domarchi (University of Southampton) - Presenting Author
Dr Olga Maslovskaya (University of Southampton)
Professor Peter Lynn (University of Essex)
Professor Rory Fitzgerald (City, University of London)
Dr Nhlanhla Ndebele (City, University of London)
Dr Ruxandra Comanaru (City, University of London)
Data collection organisations are shifting toward new approaches, with social surveys undergoing significant design and implementation changes. Since the COVID-19 pandemic, agencies have increasingly moved to online data collection due to increased internet penetration and mobile ownership but also decreasing response rates and rising fieldwork costs. A key challenge for self-completion general population surveys is the absence of field interviewers to facilitate recruitment and to guide participants though the questionnaire if needed..
In this presentation, we review recruitment practices in social surveys conducted without field interviewers in the UK, with the goal of identifying methods to optimise design characteristics and to achieve more representative samples of the general population. We begin by reviewing the academic literature on methodological aspects of self-completion surveys that use address-based sampling frames. Our review specifically focuses on recruitment practices as well as on literature on underrepresented populations, response rates, and survey costs. A total of 211 academic papers meeting the search criteria are included in our review.
We supplement this review with an information acquired from the UK’s nine leading survey agencies, which have experience in conducting surveys without field interviewers between 2018 and early 2024. The information received includes technical and methodological reports, as well as communication materials. This evidence review encompasses 144 instances of 60 longitudinal and cross-sectional surveys, along with 227 pieces of communication materials.
To our knowledge, this review is the first coordinated effort to collate and summarise recruitment strategies for surveys conducted without field interviewers. It covers sampling design, communication strategies and materials, incentivisation, fieldwork procedures, response rates, and quality assessments of different reports. This work offers insights into the current state of survey practice in recruitment of survey participants and helps identify approaches that may contribute to higher response rates and improved sample.
Mr ihsan kahveci (University of Washington) - Presenting Author
Dr McKenna Parnes (University of Washington)
Dr Brittany Blanchard (University of Washington)
Dr Tyler McCormick (University of Washington)
The U.S. overdose crisis claimed over 107,000 lives last year, with nearly 20 million Americans at risk due to fentanyl-contaminated drug supplies. Proven public health interventions, such as overdose education, naloxone distribution, and drug testing, can prevent overdose deaths. Unfortunately, access to these programs is often hindered by social stigma and systemic inequities. A promising solution lies in leveraging the social networks of people who use drugs (PWUD). Research shows that trusted peers can effectively disseminate information and resources, improve access to services, promote safer behaviors, and reduce stigma. However, traditional network analyses require complete data, often infeasible due to recruitment challenges and high costs.
This study leverages Aggregate Relational Data (ARD) to explore the social connections of PWUD and estimate key network characteristics, such as degree and centrality. Through social media advertisements on Meta, 3,600 participants were recruited to provide relational data by answering questions such as, “How many people do you know with a given trait X?” Data were collected on both close friendships and broader acquaintance networks, providing a comprehensive view of social connections within this population. The Network Scale-Up Method (NSUM) was applied to ARD, estimating an average degree of 2.7 for close friendships and 35.6 for acquaintances. Additionally, ARD methods were used to model network features and identify key actors positioned to facilitate harm reduction interventions, such as naloxone distribution and overdose prevention education. To address selection bias and ensure the sample accurately represents the target population, we applied post-stratification and design weights, calibrated using demographic data from the U.S. Census and Meta Audience Insights for Washington State.
These findings highlight the potential of ARD to collect behavioral and network data from hard-to-reach populations for designing effective public health interventions.