All time references are in CEST
Recruiting Web Surveys via Postal-Mail: Best-Practice, Experiments, and Innovation 1 |
|
| Session Organisers |
Dr Jean Philippe Décieux (Federal Insitute for Population Research) Dr Carina Cornesse (DIW Berlin) |
| Time | Tuesday 18 July, 11:00 - 12:30 |
| Room | U6-01a |
Since e-mail addresses are usually unavailable on standard sampling frames of broader population surveys (e.g., population registers), recruiting high-quality web surveys is challenging. When conducting such large-scale and large-scope web surveys, recruitment and surveying is, therefore, typically conducted in two separate steps: First, a (probability-)sample of the study population is drawn and contacted offline, often during a brief face-to-face or telephone recruitment-interview. Second, members of the sample are asked to switch to the online mode for the actual survey.
Compared to interviewer-administered contact and recruitment, postal-mail strategies are becoming increasingly popular and a large number of cross-sectional as well as longitudinal web survey projects are currently being initiated using postal-mail recruitment in combination with online survey methodology. There are several reasons for this. For example, recruiting web surveys via postal-mail is usually both more time- and cost-efficient than the available alternatives. In addition, this strategy avoids undesirable interviewer effects and allows respondents to read through study and recruitment material at their own speed, time, and convenience.
Currently, the methodology for successful postal-mail recruitment of web surveys is advancing fast. Therefore, this session aims to provide a broad exchange forum for researchers and projects working on and with postal-recruited web surveys. In addition to sharing experiences and best-practices, we are particularly interested in experimental approaches that might include, topics such as:
• Strategies for enabling the transition from offline contact to web data collection mode
• Comparing the success of postal-mail recruitment to other web survey recruitment strategies
• Optimizing initial response, panel consent, and panel registration for postal-mail recruited longitudinal studies
• Push-to-web and other mixed-mode recruitment approaches
• Cost-benefit analyses of different incentive and reminder strategies
• Design and layout effects
Keywords: Web Survey; Recruitment; Mixed-Mode; Survey Costs; Postal Recruitment; Experimental survey research
Dr Markus Hahn (Australian National University) - Presenting Author
Professor Nicholas Biddle (Australian National University)
This paper presents findings from recruiting a small probability-based online panel, a pilot we conducted to support the design of a larger panel we plan for 2024. Trialling a dual-frame sampling strategy, while also conducting some recruitment experiments, we examine initial response outcomes and subsequent attrition for two recruitment modes—mail and SMS. Using data from four waves, we are able to track the characteristics and outcomes of these two samples across time.
A probability sample of 346 adults was recruited from the Australian electoral roll (through mail); another sample of 272 adults from the population of Australian cellphone owners (through SMS text messages). Both samples were given the same online questionnaire, initiated through push-to-web. Incentives were offered upon questionnaire completion. The mail sample also received an unconditional incentive with their invitation. Respondents enrolled in the panel after completing wave 1. Contacts in subsequent waves occurred through email.
Mail-based recruitment generated superior response rates compared to SMS-based recruitment (12.8% vs 1.4%). Socio-demographic characteristics of both samples were similar, with some exceptions. The mail sample better captured those with lower education; the SMS sample better captured younger persons. Panel attrition was lower among participants recruited through mail.
An experiment varied the recruitment message shown to potential participants. Both samples were shown the same text, but through different modes. The experiment varied the letter for the mail sample and the website for the SMS sample. The letter generated more variation across treatments, highlighting the difficulty of delivering detailed information through SMS rather than mail. The SMS sample spent more time completing the online questionnaire and was also more likely to abandon an incomplete questionnaire. This suggest there is less friction in push-to-web from mail than from text messages received on cellphones.
Dr Raül Tormos (Centre d'Estudis d'Opinió - Generalitat de Catalunya) - Presenting Author
Dr Jordi Muñoz (Centre d'Estudis d'Opinió)
Probability-based online panels are currently a cost-effective and technically advisable survey design strategy both for official public institutions and academic research centers alike. The Center for Opinion Studies of the Catalan Government in Spain designed and implemented an infrastructure to allow the transition from face-to-face cross-sectional surveys to an online probability-based panel. Its previous survey series are now being integrated into this online platform. The first wave of the panel survey has been specifically designed with experiments for testing crucial survey design decisions that would allow undertaking the optimal choices in future waves. One of the experiments aimed at identifying the most cost-effective system of incentives for promoting respondents’ participation and sample representativity. The design of that experiment implies that 40% of the theoretical sample is assigned to a non-incentive group, another 40% goes to a conditional 5€ incentive, and the remaining 20% is invited to take part in a lottery. The group with a 5€ conditional incentive is evenly divided in one that receives the announcement of the incentive on the initial invitation, and another that receives it in the reminder. Our focus is both on the consequences of incentives for response rates and the representativity of the sample obtained. Aside from testing two types of incentives in a specific national context (Spain and Catalonia, particularly), the governmental nature of the institution in charge of fielding the survey is of a different kind to those usually studied. The theoretical sample of respondents randomly extracted from the census register receives a letter with the official stamp of the public administration requesting the potential respondents to take part in the survey. This might increase response rates but an also introduce different sorts of biases.
Mr Johannes Lemcke (Robert Koch-Institut) - Presenting Author
Mr Michael Lange (Robert Koch-Institut)
Mr Robin Houben (Robert Koch-Institut)
Mr Marcel Hintze (Robert Koch-Insitut)
Mr Ilter Öztürk (Robert Koch-Insitut)
Mrs Jennifer Allen (Robert Koch-Institut)
Mr Patrick Schmich (Robert Koch-Institut)
Background
The Robert Koch Institute (RKI) is currently implementing a panel infrastructure ("RKI Panel") geared to the needs of public health research, consisting of extensive probability and non-probability samples. The main data collection mode will be the online mode (CAWI) in order to fully exploit the speed advantages of this mode, especially for ad hoc studies.
The aim of the first recruitment phase in 2023 is to initially recruit 30,000 active (i.e. registered) panelists. The recruitment will start in march 2023.
Methodology
For the first recruitment study, the registries of the population registration offices will be used as a sampling framework to draw an address sample of persons. A sequential mixed mode design (push-to-web strategy) will be used in the age group 16 to 69 years (invitation letter with online mode-only; two postal reminders; offline mode with paper questionnaire in the 2nd reminder letter). In the age group 70+ years, a simultaneous mixed-mode design is implemented (online mode + paper questionnaire in the invitation letter). For all addresses, depending on outcome rates and the current sample composition, an optional contact phase (telephone contact and home visits for a random sample of addresses) will be implemented for addresses with previously unresolved status.
An unconditional incentive of 5€ cash will be sent with the invitation letter to all invitees. For successful registration as a panelist (online and offline respondents), an additional conditional 10€ incentive (cash payment) will be sent out. In addition, an "early bird" incentive will be paid for respondents who register as a panelist within the first two weeks (15€ instead of 10€ cash payment).
Results
The study design will be presented and first outcome rates and metrics (initial response rate, registration rate, sample composition etc.) of the first tranches.
Mr Bas Weerman (University of Southern California)
Dr Marco Angrisani (University of Southern California)
Ms Tania Gutsche (University of Southern California) - Presenting Author
The Understanding America Study is a probability-based internet panel of 10,000 respondents living in the United States, recruited by mail to take surveys online. The return to address-based sampling as the gold standard has commonly been accepted, but low response rates plague U.S. panels. Our study began in 2014 and has instituted a number of experiments since inception. In 2014 33% of our initial 10,000 approached returned the paper survey and consented to join our study, 19% actually because active (took our first two surveys online). In 2022, on average 14.5% were interested in joining, and 9.75% actually joined. What has changed, and why are so many of our $5 bills just going to the landfills?
We follow a recruitment approach based upon the Dillman method, thus always including a prepaid incentive of $5, but have continue to explore ways to reach diverse groups and increase response rates to our initial paper survey. These variations include changes in envelopes including visible cash, priority mail and first-class mail, personalized and generic greetings, push to web or paper first, and return postcards. We have experienced declines in consent as trust of the internet wanes and privacy fears wax; ironic in that the protections our online platform offers far exceed those consented to in social media and other commonly used applications. Through this paper we present an overview of the challenges faces survey researchers today, the quest to imbue trust in potential respondents, and our most successful experiments in initial survey returns and those consenting and remaining active in our panel.
Dr Christiane Bozoyan (LMU Munich) - Presenting Author
Dr Claudia Schmiedeberg (LMU Munich)
Dr Jette Schröder (GESIS)
For achieving high response rates in a web survey, potential technical or organizational barriers to participation should be removed to make participation as easy as possible. Therefore, invitation letters via postal mail often include a QR-Code in addition to the web-address where the survey can be started so that respondents need not type the web-address and their personal code.
However, as QR-codes are usually used with mobile devices (i.e., smartphones and tablets) but not desktop or laptop computers, a QR-code facilitates participation especially via mobile devices and may thus affect which devices will be used for taking the web survey. This may be a drawback if web surveys are less comfortable if completed on small screens as in these cases a QR-code might increase the risk of break-up and compromise data quality (e.g., satisficing, incorrect entries, item nonresponse).
We test effects of a QR-code in the invitation letter in a push-to-web survey making use of an experiment in wave 14 of the German Family Panel pairfam. One randomly drawn half of the sample received a letter with a personalized QR-code in addition to the URL and the personal code whereas the other half received only the URL and the personal code. We investigate whether the QR-code increases participation as well as the devices used for completing the survey.