Challenges and Opportunities of Switching to Web 1 |
|
| Session Organiser |
Dr Susana Cabaço (Netherlands Interdisciplinary Demographic Institute) |
| Time | Tuesday 16th July, 11:00 - 12:30 |
| Room | D18 |
This session includes papers that address the challenges and opportunities of switching survey designs to web-based data collection, particularly in push-to-web mixed mode designs.
Keywords: mixed mode, push-to-web, web first, mode effects
Dr Rachel Horwitz (U.S. Census Bureau) - Presenting Author
Ms Beth Newman (U.S. Census Bureau)
Mr Joshua Neufelder (U.S. Census Bureau)
Mixed-mode designs are becoming more common as surveys combat rising costs and declining response rates. Adding an additional data collection mode can have both positive and negative effects. For example, adding a web response option can lead to shorter response times, less data processing, the elimination of errors due to unclear skip patterns, and reduced item nonresponse from edit checks. On the other hand, there are well-documented measurement effects that result from mixing modes that can impact survey estimates and inferences that rely on response data. Much of this research is on the general population or of subsets of the general population (i.e., college graduates, students, people with children, etc…). The research presented here discusses the results of an experiment of schools where one group received an internet first methodology and the other received only a paper questionnaire. We will compare overall response rates, response rates by mode, the types of schools that used each mode, and the timing of response by mode. Here we will be able to evaluate whether a school behaves similarly to individual respondents. For example, most principals have a computer at their school whereas more members of the general public do not have access to one. Additionally, principals receive an invitation to complete the survey at their place of business and where that computer is located. Does this mean they will respond online at a higher rate than other populations? Alternatively, given they likely need to collect information from multiple sources to complete the survey, will response times for web not decrease as the research suggests for person- or household-based surveys? This research will shed light on the impact of adding an internet response mode to a school survey and how they may differ from traditional population surveys.
Dr Vidal Díaz de Rada (Public University of Navarre) - Presenting Author
The proposed study confirms the finding that the sequential use of three different modes of data collection increases research quality, as it improves response rates, reduces measurement error, and decreases coverage error. It also reduces collection time and cost. This has been confirmed by the use of a sequential mixed design and three modes of data collection: online, telephone and face-to-face surveys.
Fieldwork began by using an online channel that yielded response rates of 39%. After 6-8 weeks, a letter was sent to those who had not responded informing them that a telephone survey would be conducted, which was answered by 47% of those contacted. Those who failed to respond were them interviewed by telephone, which yielded a 50% response rate. The people whose telephone number was not available were contacted again in person. This resulted in a final sample of 7,255 participants: a total of 44.3% provided a self-administered response, 44% were interviewed by telephone, and 11.7% responded in a face-to-face interview.
Out of the numerous aspects that may be analysed concerning the advantages of mixed modes, this study focuses on obtaining the best access to specific groups, decreasing research costs, and improving the quality of the data collected. Both the sociodemographic features and the subject of research are considered with regard to accessing ‘new’ population groups. Within these groups, the online survey was answered fundamentally by the most dynamic sectors of the sample universe (employees and students), while retired people and those who worked at home largely responded to face-to-face and telephone surveys.
Dr Susana Cabaço (Netherlands Interdisciplinary Demographic Institute) - Presenting Author
Ms Judith Koops (Netherlands Interdisciplinary Demographic Institute)
Dr Tom Emery (Netherlands Interdisciplinary Demographic Institute)
Dr Peter Lugtig (University of Utrecht)
Dr Vera Toepoel (University of Utrecht)
Dr Detlev Lück (Federal Institute for Population Research)
Ms Almut Schumann (Federal Institute for Population Research)
Mr Robert Naderi (Federal Institute for Population Research)
Dr Martin Bujard (Federal Institute for Population Research)
The Generations and Gender Programme (GGP) is a cross-national research infrastructure which aims at understanding family and life course dynamics. In 2018, the GGP conducted a pilot study in Germany, Croatia, and Portugal to investigate whether using a web first design would be suitable for future waves. One concern with the implementation of the CAWI mode was the level of break-offs and how they were distributed across type of device given the rising prevalence of smart phones and other devices. The Generations and Gender Survey was being has a long and complex structure but is also historically established and so cannot be fully adapted for web. This can introduce challenges to respondents using devices with smaller touchscreens like smartphones.
In our analysis we will be focusing on survey paradata collected to understand the patterns in break-offs across the three countries. The paradata was collected for 3,237 web interviews using Blaise 5.3 and includes information on the device and browser type used. Of these, approximately 16% of interviews broke off before the core modules of the questionnaire were completed. The central objective in this analysis is to explore to what extent break-offs and respondent behaviour were influenced by the device used. The countries involved in the study have variable rates of internet and smart phone penetration and so also offer an opportunity to assess whether the role of specific devices in breaks off varied across different contexts. In all three countries the sample was stratified by urban/rural residence, allowing for comparisons across 6 distinct contexts in total. The lessons from this work will inform future strategies to tackle high break-off rates and the anticipated uptake of various devices in populations with relatively low internet penetration.
Dr Detlev Lück (Federal Institute for Population Research (BiB))
Mrs Almut Schumann (Federal Institute for Population Research (BiB)) - Presenting Author
Mr Robert Naderi (Federal Institute for Population Research (BiB))
Dr Martin Bujard (Federal Institute for Population Research (BiB))
Professor Norbert Schneider (Federal Institute for Population Research (BiB))
The paper presents results from a push-to-web experiment, conducted 2018 in the context of the Generations and Gender Survey (GGS). GGS is a panel study focussing on family-demography. It has a questionnaire of approximately 60 minutes length with complex routing. In the past, GGS has been conducted in CAPI mode. An experimental pilot study, conducted in three countries (Germany, Croatia, Portugal), has investigated whether GGS could move to a mixed-mode design and which design would be recommendable to do so. For this purpose, CAPI and a sequential mixed-mode (push-to-web) design, combining CAWI and CAPI, have been compared in the three country contexts. In each country, an additional experiment has been conducted.
This paper presents the results of the German sub-experiment where five strategies of offering incentives have been compared. The incentives were: 1) no incentive, 2) 5 Euro unconditionally pre-paid, 3) 5 Euro conditionally post-paid, 4) 5 Euro pre-paid plus 25 Euro post-paid, and 5) 30 Euro post-paid. Despite evidence from past experiments, generally recommending a small unconditional incentive, a GGS-specific experiment seemed necessary since the GGS questionnaire is exceptionally long for CAWI mode. Therefore it seemed possible that a higher – and therefore eventually (partly) conditional – incentive could work better in the particular context.
Since fieldwork is on-going on the day of submission, results reported here are preliminary, but likely to hold true: Response rates can clearly be increased with 5 Euro unconditionally. However, a high incentive works even better. The combined incentive (5 + 25 Euro) works best. The generally high break-off rates can be reduced by incentives accordingly. Effects on data-quality have not been investigated yet. What can be increased rather unsatisfactorily by incentives is the consent to store contact information for a re-contact, which is generally low in CAWI mode.