All time references are in CEST
Recruiting Web Surveys via Postal-Mail: Best-Practice, Experiments, and Innovation |
|
| Session Organisers | Dr Jean Philippe Décieux (University of Bonn & Federal Institute for Population Research (BiB)) Dr Carina Cornesse (GESIS) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Since e-mail addresses are usually unavailable on standard sampling frames of broader population surveys (e.g., population registers), recruiting high-quality web surveys is challenging. When conducting such large-scale and large-scope web surveys, recruitment and surveying is, therefore, typically conducted in two separate steps: First, a (probability-)sample of the study population is drawn and contacted offline, often during a brief face-to-face or telephone recruitment-interview. Second, members of the sample are asked to switch to the online mode for the actual survey.
Compared to interviewer-administered contact and recruitment, postal-mail strategies are becoming increasingly popular and a large number of cross-sectional as well as longitudinal web survey projects are currently being initiated using postal-mail recruitment in combination with online survey methodology. There are several reasons for this. For example, recruiting web surveys via postal-mail is usually both more time- and cost-efficient than the available alternatives. In addition, this strategy avoids undesirable interviewer effects and allows respondents to read through study and recruitment material at their own speed, time, and convenience.
Currently, the methodology for successful postal-mail recruitment of web surveys is advancing fast. Therefore, this session aims to provide a broad exchange forum for researchers and projects working on and with postal-recruited web surveys. In addition to sharing experiences and best-practices, we are particularly interested in experimental approaches that might include, topics such as:
• Strategies for enabling the transition from offline contact to web data collection mode
• Comparing the success of postal-mail recruitment to other web survey recruitment strategies
• Optimizing initial response, panel consent, and panel registration for postal-mail recruited longitudinal studies
• Push-to-web and other mixed-mode recruitment approaches
• Cost-benefit analyses of different incentive and reminder strategies
• Design and layout
Keywords: Web Survey, Recruitment, Mixed-Mode, Survey Costs, Postal Recruitment, Experimental survey research
Ms Tania Gutsche (University of Southern California) - Presenting Author
Mr Bas Weerman (University of Southern California)
Ms Jenna Blyler (University of Southern California)
Dr Arie Kapteyn (University of Southern California)
Approached by colleagues at USC, the Center for Economic and Social Research (CESR) identified members of our probability based national internet panel, The Understanding America Study (UAS), who were living adjacent to an aerospace lab that has been in operation under numerous owners since 1955, in the Los Angeles region. We then complemented our known sample with an ABS push-to-web approach. Though our center has numerous experiences with specific population recruiting, we had always included a letter of institutional support, e.g. from the Mayor (Compton Well-Being Study) or the Los Angeles Department of Public Health (DPSS Tayportunity Study). This survey was solely researcher-driven. The 10-minute survey offered at $20 Amazon gift code, and landed with a population keen to tell their story. Our colleagues had done a great deal of research in the field to make sure the survey was relevant through in- person interviews and community meetings. The survey explored topics such as knowledge of environmental effects, correct assignment of blame/clean up responsibility, duration of time living near the site, intent to move, and self-reported health outcomes such as cancers for themselves and their pets. At the end the survey offered an opportunity to further share their story with a qualitative researcher by leaving their contact information. We achieved a response rate of 28%, which is higher than our prior California ABS recruitment rates. Contributing factors outside of familiarity or trust in our local academic institution are likely in part due to the immediacy of the topic. Through this experiment, we are encouraged to find a way to make our ABS recruitment campaigns as personally engaging as this one.
Mr Felix Cassel (University of Gothenburg) - Presenting Author
Dr Sebastian Lundmark (University of Gothenburg)
Rising costs of traditional survey modes and declining response rates are well-documented challenges for contemporary survey research. These challenges may have implications for the representativeness of survey results. Push-to-web data collection methods are increasingly used by survey practitioners to decrease the costs of printing and postage when mailing back physical questionnaires. This study examined whether two different push-to-web strategies produced dissimilar survey response rates, nonresponse bias, and subsequent willingness to join an online access panel.
In 2024, the SOM Institute at the University of Gothenburg conducted a push-to-web mixed-mode survey of a random probability sample of the Swedish population (N = 4,046). The sample persons received four invitations by mail and two text message reminders. Prior to inviting the potential respondents, all sample persons were randomly assigned into two experiment groups: 1) a “soft-push-to-web” group (n = 2,027) in which the participants received a paper questionnaire by mail from the first reminder, and 2) a “hard-push-to-web” group (n = 2,019) in which a paper questionnaire was distributed by mail only in the third and final reminder. At the end of the survey, participants were asked to sign up for the Swedish Citizen Panel (SCP), a non-commercial online access panel administered by the SOM Institute.
Preliminary findings from the experiment showed that the softer-push-to-web recruitment strategy increased response rates by approximately 4 percentage points. Further analyses on the recruitment strategy’s impact on nonresponse bias as well as the likelihood of joining the SCP will be conducted in early 2025 and presented at the conference.
Mrs Martha McRoy (NORC at the University of Chicago) - Presenting Author
Mr Ned English (NORC at the University of Chicago)
Mr Ben Reist (NORC at the University of Chicago)
Ms Amie Conley (NORC at the University of Chicago)
Ms Samantha Saini (Illinois Department of Public Health)
Ms Kelsey Cutler (Illinois Department of Public Health)
Survey researchers continue to face an environment characterized by falling response rates, even when using address-based samples with multi-mode, web-push designs. Recruitment materials for web-push surveys in the United States are typically mailed using standard white #10 envelopes and may include a prepaid incentive. Our process raises two questions: a) Would more people open a different envelope, increasing participation and b) does raising the prepaid incentive from $1 to $2 mitigate nonresponse in lower-performing areas? We designed experiments to explore alternatives to the usual envelope and increased incentives to answer these questions.
The Healthy Illinois Survey is a sequential mixed-mode survey (web-push, inbound CATI, PAPI follow-up) conducted on behalf of the Illinois Department of Public Health. Sampled households received an initial invitation letter and a reminder postcard. Then, for nonresponding households, a third mailing is sent. In our envelope experiment, households were randomly assigned to receive the third mailing as either (1) a letter in a standard #10 white envelope, (2) a letter in a brown kraft #10 envelope, (3) a letter in a white 9x12 envelope, (4) a letter in a white 6x9 envelope, or (5) a packet including a paper questionnaire.
Initially, all households received a $1 prepaid incentive in their invitation letter. Underperforming strata (areas with lower response) had their incentives increased to $2 in the later waves to help boost participation.
We examine the response rates and cost per complete by mailing type to assess if a different envelope is optimal for nonresponse follow-ups and the impact of the raised incentive. We analyze demographic characteristics and key substantive items for respondents to the different conditions to see if one envelope style or amount may better reach harder-to-respond populations and improve representation.
Mr Georg-Christoph Haas (Institute for Employment Research) - Presenting Author
Mrs Marieke Volkert (Institute for Employment Research)
Mr Stefan Zins (Institute for Employment Research)
For online probability surveys that recruit participants via postal invitation letters, passwords are used to manage access to the survey. These passwords serve several purposes, such as, blocking uninvited individuals, and preventing multiple submissions from the same individual. Research on web survey passwords has primarily focused on whether providing a password for survey access affects response rates. However, the chosen password strength, that is, length and complexity, may also affect response propensities. Password length refers to the number of characters in a password. Password complexity involves the set of characters from which the password can be derived (e.g., lowercase letters and numbers). Our research evaluates the effect of different password lengths and complexities on survey access, response rates, panel registration and linkage consent. We implemented a survey experiment by varying password length and complexity during the first wave of the OPAL study. For recruitment, every individual received a postal invitation letter with a web-link and QR-code directing to the survey, along with an individualized password. We conducted a 2×2 experiment that manipulated password length (five and eleven characters) and complexity (uppercase letters and uppercase + lowercase letters + numbers). Additionally, we included a group that used the default length and complexity settings of the service hosting the survey (eight uppercase letters). Invited individuals were randomly assigned to one of these five groups across two different probability samples: employees (N=77.173) and welfare recipients (N=99.176). Results show that short as well as long passwords increase the access rate compared to the control group (16.7%, 19.2% vs. 14.9%). The positive effects of the password designs remain for response and panel registration rates. We also find that long passwords have a positive effect on the propensity to consent to linking survey with administrative data.
Dr Andreas Ette (Federal Institute for Population Research) - Presenting Author
Dr Jean Philippe Décieux (Federal Institute for Population Research)
Following Russia’s invasion of Ukraine in early 2022, more than one million refugees arrived in Germany. These Ukrainian refugees differ in many aspects from Germany’s past experiences with forced migration, creating an urgent need for reliable data to inform policymakers, practitioners, and academics. In response, the “Refugees from Ukraine in Germany (IAB-BiB/FReDA-BAMF-SOEP Survey)” was established to provide high-quality longitudinal data.
This presentation offers insights into the six waves of this new refugee panel, providing a unique case study on the challenges and strategies involved in maintaining an offline-recruited online-panel of refugees. Initiated during the early months of the war in Ukraine, the survey employed a register-based sampling frame to recruit participants offline, who were then invited to participate in subsequent survey waves online. This hybrid recruitment approach aimed to balance the need for representative sampling with the flexibility of digital data collection, particularly for the highly mobile Ukrainian refugee population.
The presentation evaluates the sustainability of the offline-recruited online-panel in two steps. First, the determinants of nonresponse in the recruitment wave are analyzed using auxiliary personal information from Germany’s population register, alongside detailed regional and survey characteristics. This highlights the key factors influencing participation, attrition, and engagement. Second, the determinants of retention—stayers, gradual attriters, and attriters—are examined across six panel waves, demonstrating how survey design decisions in the recruitment wave have lasting impacts on panel participation.
We explore how initial offline recruitment shaped panel composition and how digital communication strategies, such as email reminders and adaptive fieldwork techniques, supported retention. Special attention is given to subgroup-specific dynamics to identify disparities in participation patterns. The findings underscore the importance of integrating adaptive strategies into survey design to sustain engagement in online panels over time.
Mr Ned English (NORC) - Presenting Author
Mr Phil Schumm (The University of Chicago)
Ms Evgenia Kapousouz (NORC)
Ms Kate Lefauve (NORC)
Dr Colm O'Muircheartaigh (The University of Chicago)
The primary purpose of the SAMLAP* study was to recruit and interview a probability sample of the older LGBT population in the United States, following a multi-mode address-based (ABS) sample design. We present results from the recruitment phase, which administered an eligibility screener to a probability sample via different self-administered questionnaire modes. In so doing we conducted a series of experiments to test the effectiveness of different recruitment strategies, including: 1) varying incentive amounts and timings, 2) including an SAQ with the first web-push mailing, and 3) implementing a last-ditch contact via FedEx mailer. This paper will examine the costs, relative yields, and respondent characteristics for each mailing and incentive treatment. In so doing we will make recommendations on designs in situations and for specific socio-economic and ethnoracial subpopulations i.e., reducing non-response bias, increasing yields, and optimizing efficiency.
Our results will be of interest to researchers and practitioners studying sexual orientation and gender identity (SOGI), LGBT and older adults, as well as survey methodologists focused on recruitment strategies for rare populations in an era of escalating costs and decreasing response rates.
*Sampling Strategies and Measurement Approaches for the LGBT Aging Project: a feasibility study for a potential national panel of older LGBT US population; research reported in this publication was supported by the National Institute on Aging of the National Institutes of Health under Award Number R01AG070059
Professor Nicole Gürtzgen (IAB)
Dr Alex Kübis (IAB)
Dr Andre Pirralha (IAB) - Presenting Author
As web-based data collection continues to replace traditional interviewer-led modes, effective offline-to-online recruitment has become central to large-scale surveys. This presentation evaluates the first year of a push-to-web mail recruitment strategy in the German Job Vacancy Survey (IAB-JVS), a repeated cross-sectional establishment survey that draws a large, stratified random sample of about 110,000 German establishments each year.
Traditionally, the IAB-JVS included a comprehensive questionnaire in the fourth quarter, complemented by shorter telephone surveys in subsequent quarters. Recently, the design shifted toward inviting establishments to complete the survey online, relying on postal mailings as the primary contact. If no response is received, a follow-up mail reminder includes an option to participate via a paper-and-pencil questionnaire.
This push-to-web approach offers an interesting setting to examine how postal mail can facilitate transitions from offline contact to web-based participation, especially in a survey that must balance cost, large sample sizes, and diverse establishment types. Drawing on data from the first year of implementation, we assess key metrics such as response rates, fieldwork timing, and nonresponse patterns. The findings contribute to the discussion on postal-mail recruiting strategies by illustrating both strengths and limits in reaching certain subgroups and maintaining data quality. By sharing these insights, we aim to inform best practices for recruiting web surveys via postal-mail, particularly in the context of establishment surveys.
Dr Ferenc Mújdricza (Hungarian Central Statistical Office)
Mrs Mária Zanatyné Fodor (Hungarian Central Statistical Office)
Mr Mátyás Gerencsér (Hungarian Central Statistical Office) - Presenting Author
Mrs Linda Mohay (Hungarian Central Statistical Office)
The first impression on most household survey respondents, advance letter design, is paramount in gaining cooperation – especially in online self-completion. The main approaches, the ‘old’ formal, mandatory approach and the ‘new’ respondent-centred approach with behavioural science techniques were combined at the Hungarian Central Statistical Office. This ‘new-old’ official design employs authoritative tone, purposefully complicated text, legal references, pressing nudges, austere graphics, etc. in a tweaked behavioural science framework so as to harness the potential of the mandatory approach in voluntary participation surveys. Initial experiments proved that the ‘new-old’ formal design performs drastically better than semi-formal alternatives. Priority mailing added further significant amplification to the effect of the formal design.
The two split-sample experiments in the present study with further improved push-to-web letter designs and additional postage modes of formal letters reinforced previous results in a CAWI setting. The authoritative/formal design yielded significantly better response rate and response quality than a semi-formal letter, tailored to target young respondents, as well as a novel ‘honest’, trust-inducing design. The latter was to inform in a candid, open way, avoid manipulative features, and express trust by sharing facts (e.g. out-of-date databases necessitate conducting the survey). Priority mailing of formal letters increased cooperation significantly in comparison with the ordinary mail condition, with the additional benefit of more prompt responding. Registered postage performed similarly, but with much higher unit costs.
In sum, thinking more ‘inside our box’ is useful in designing effective push-to-web advance letters: in contrast with semi-formal, ‘respondent centred’ letters, formal advance letter designs harvest core, official characteristics of NSOs. This authenticity might be essential in gaining cooperation as it probably aligns better with general respondent expectations towards (statistical) offices, superseding respondent ‘preferences’ or ‘needs’.
Mr Stefan Damerow (Robert Koch-Institut) - Presenting Author
Mr Johannes Lemcke (Robert Koch-Institut)
Mr Ilter Öztürk (Robert Koch-Institut)
Mrs Jennifer Allen (Robert Koch-Institut)
Background
The Robert Koch Institute (RKI) set up a probability-based panel infrastructure focused on public health research (‘Health in Germany’) (registered active panelists about 47.000). Due to declining response rates in recent decades, incentives have become increasingly important. Therefore, for the first recruitment study of the panel Health in Germany the RKI conducted an incentive experiment with a random sub-sample in order to test the effectiveness of different incentive schemes. The central questions of the incentive experiment are: Differentiated by incentive group (a) the response rate for panel registration, (b) the possible distortion due to non-response bias and (c) the panel case costs.
Methodology
The study population comprises all persons aged 16 and over living in Germany. Around 170,000 addresses of the residents' registration offices were used as a random sample. The field period ran from January to May 2024. Some of the target persons were randomly selected for an incentive experiment with four groups of 1440 individuals each. The incentives were either paid unconditionally beforehand (‘before’) or were linked to registration for the panel (‘after’). The incentive schemes of the groups were: (1) €5 before, €10 after, in cash; (2) €10 after, in cash; (3) €5 before, €10 after, as a voucher; (4) no incentive at all (control group).
Results
The incentive experiment replicates existing incentive experiment studies. The use of cash instead of vouchers and the unconditional payment (‘in advance’) of €5 substantially increases the willingness to participate (about 12 percentage points difference). Particularly in the hard-to-reach population group of people with a low level of education, a less biased sample composition can be observed in comparison to all other incentive groups. The results of the incentive experiment provide insights into how high data quality can be achieved with fewer resources.
Dr Detlev Lück (Federal Institute for Population Research (BiB)) - Presenting Author
Dr Pablo Christmann (GESIS - Leibniz Institute for the Social Sciences)
Dr Tanja Kunz (GESIS - Leibniz Institute for the Social Sciences)
Professor Tobias Gummer (GESIS - Leibniz Institute for the Social Sciences, Germany)
Ms Angelika Steinwede (infas - Institute for Applied Social Science, Germany)
“FReDA - The German Family Demography Panel Study” is a large German panel study that fielded its recruitment wave in Spring 2021. Interviews are conducted in self-administered modes: mainly by web interviews (CAWI), partly by mail-sent paper questionnaires. The initial plan to switch to self-administered modes only after a personal recruitment interview was disrupted by the COVID-19 pandemic. So, also the recruitment took place by postal invitation letters and self-administered interviews. This was accomplished with satisfying rates of participation and panel consent. One presumably helpful design feature was a short recruitment questionnaire with low respondent burden. A second feature is an unconditional prepaid incentive of 5 Euro in cash that is added to each invitation, in the recruitment as well as in all subsequent waves.
In late 2024, a first refreshment sample was drawn. Although there are no longer any contact restrictions, FReDA still does not use interviewers and continues with its recruitment strategy described above. Nevertheless, the recruitment of the refreshment came with a number of experiments which are supposed to help optimising this strategy. In a first experiment, the gross sample (n=50,270) was split up into one experimental group interviewed in web mode only and a second group interviewed in a sequential “web first” mixed mode design. Further experiments tested the effects of various enclosures to the invitation letter: a project information flyer, a greeting card that plays a sound file when opened, and a card with a QR code inviting to visit an online motivation video. One experiment tested the effect of sound files in the web questionnaire, reading out the questions. Finally, the effect of an additional 10 Euro postpaid inventive in case of a panel consent was tested. We compare outcome rates for participation, panel consent and sample composition.