All time references are in CEST
Push-to-Web Surveys: Challenges and Opportunities 3 |
|
| Session Organisers | Dr Olga Maslovskaya (University of Southampton) Dr Peter Lugtig (Utrecht University) |
| Time | Thursday 20 July, 14:00 - 15:30 |
| Room | U6-23 |
We live in a digital age with widespread use of technologies in everyday life. Technologies change very rapidly and affect all aspects of life, including surveys and their designs. Online data collection is now common in many countries and some surveys started employing push-to-web approach (for example, Generations and Gender Surveys in some countries, including the UK) in which offline contact modes are used to encourage sample members to go online and complete a web questionnaire. This method of data collection is typically used when sampling frames do not include email addresses for all members of the target population. It is important to address different methodological challenges and opportunities which are associated with push-to-web surveys.
This session welcomes submissions of papers on different methodological issues associated with pish-to-web surveys in both cross-sectional and longitudinal contexts. Examples of some topics of interest are:
• Coverage issues
• Data quality issues
• Unit nonresponse
• Response rates and nonresponse bias
• Mobile device use
• Questionnaire length
• Other topics
We encourage papers from researchers with a variety of backgrounds and across different sectors, including academia, national statistics institutes, data collection organisations, and research agencies.
This session aims to foster discussion, knowledge exchange and shared learning among researchers and methodologists around issues related to push-to-web surveys. The format of the session will be designed to encourage interaction and discussion between the presenters and audience.
Keywords: online surveys, push-to-web surveys, nonresponse bias, response rates, data quality
Ms Alex Bogdan (Ipsos UK) - Presenting Author
The UK has one of the highest internet penetrations in the world, with the ONS estimating that 96% of all households in the UK had internet access in 2020. With the Covid 19 pandemic forcing many to communicate with loved ones online for extensive periods, the expectation is perhaps that we are now an online society. With 89% of all responses to the 2021 Census in England and Wales being online completions, is there any reason to offer offline modes of completion to respondents to push-to-web survey or can we direct resources elsewhere?
We argue that internet access is not the only barrier to online survey completion in the UK and that internet confidence is a more relevant predictor of survey completion mode.
Our presentation will use data from the Ipsos UK KnowledgePanel, an online random probability panel, which uses a postal push-to-web methodology to invite members of the public to complete an online survey and become panel members. The registration process is tailored to individuals’ levels of comfort of using the internet, with additional modes and layers of support built in for different needs: from limited support in completing an online survey, to returning a postal form and receiving a tailored email invitation, telephone support and finally receiving a tablet with restricted internet access. Additional support is available through a bespoke helpline available to those who are digitally excluded and through printed materials.
We will use new, unpublished data to explore online behaviours and attitudes of those who are digitally excluded, focusing particularly on confidence of using the internet. We argue that in order to achieve a representative sample, it is important to be mindful of differing levels of internet confidence when designing survey tools, while still delivering an online-only survey, avoiding mode effects.
Dr Pablo Christmann (GESIS – Leibniz Institute for the Social Sciences) - Presenting Author
Dr Tobias Gummer (GESIS – Leibniz Institute for the Social Sciences)
Dr Armando Häring (infas – Institut für angewandte Sozialwissenschaft GmbH)
Dr Tanja Kunz (GESIS – Leibniz Institute for the Social Sciences)
Ms Anne-Sophie Oehrlein (GESIS – Leibniz Institute for the Social Sciences)
Mr Michael Ruland (infas – Institut für angewandte Sozialwissenschaft GmbH)
Dr Lisa Schmid (GESIS – Leibniz Institute for the Social Sciences)
The COVID-19 pandemic impacted the operation process of many survey programs, among them the recruitment of the newly established German panel study FReDA (Family Research and Demographic Analysis). Switching from face-to-face interviews to self-administered mixed-modes (web, paper) has allowed us to experimentally test the effectiveness of different mode choice sequences. With this study, we adresss existing research gaps regarding outcome rates, data quality, and survey costs.
FReDA is based on a large register-based sample of persons aged 18-49 living in Germany. In the recruitment wave, we implemented three different experimental conditions to which individuals were assigned randomly. FReDA deploys an invitation letter and reminders that offer either an access-link/QR-Code to the web survey (Web), or containing the paper-based questionnaire and an access-link/QR-Code to the web survey (Web/Paper) in different sequences to contact the gross sample. Individuals are contacted either with:
(1) a concurrent design in the sequence Web/Paper, Web, Web/Paper, or
(2) a web-first design in the sequence Web, Web, Web/Paper, or
(3) a push-to-web design in the sequence Web, Web, Web, Web/Paper.
By design, the third condition also allowed us to simulate and compare how the recruitment would have performed in a web-only mode with one invitation letter and two reminders:
(4) web-only design in the sequence Web, Web, Web.
Based on the experimental data, we analyze how different mode choice sequences affect the response rate, number of shippings, postage, share of paper, distributions of substantive answers, sample composition, data quality, panel consent, and participation in the subsequent wave.
Our results show that the web-first design performed best. This design yielded a comparatively good response rate and recruitment rate at reasonable survey costs.
Professor Virginia Lesser (Oregon State University- Department of Statistics) - Presenting Author
Ms Lydia Newton (Oregon State University - Survey Research Center)
Ms Jessica Robinson (Oregon State University - Survey Research Center)
Over the past several years, the Web has become a popular method to collect survey data. Traditionally, mail is also used and a number of surveys used both modes to collect survey data, referred to as Web+Mail or a Web-Push. Government agencies often track opinions and patterns of behavior in repeated studies over time. Although collection of survey data using the Web can be quicker and less expensive than sending printed surveys using the postal service, studies have shown that not all individuals in the general population have access to the Internet. However, it is often the case that email addresses are not available in certain populations. In addition, Web responders are different demographically than the demographics of a general population. Prior to making a decision to only use the Web to collect survey data, studies to understand the impact on survey results due to changing the survey collection mode have been and are currently being examined. A study of boat use in the State of Oregon has been collected since 2009, with the most recent study completed in 2022. In each of the four studies discussed in this paper, split samples were designed using an All Mail and a Web+Mail mode so that the response rates and estimates of boat use by mode over time could be compared. Up to four contacts were sent to boat owners in each sample. Nonrespondents in the Web+mail mode were given the option in the final reminder of completing the survey either by Web or by mail. Responses obtained from only Web responders, the Web+Mail responders and responders in the All Mail study groups are compared.
Dr Pedro Silva (Statistics Portugal) - Presenting Author
Ms Tânia Correia (Statistics Portugal)
Ms Filipa Lidónio (Statistics Portugal)
Within a sequential-mixed mode data collection approach, a multimode contact strategy is generally used at Statistics Portugal, including for CAWI promotion in push-to-web designs. In addition to postal and email communications, text messages have also shown to be helpful in reducing overall nonresponse and increasing web responses. Its use will be discussed in the current presentation, as part of an integrated mixed-mode management adaptive strategy.
In addition to the impact of sending an additional push-to-web SMS, we focus on the different advantages it may entail (e.g., invalid numbers identification prior to CATI stage), as well as the specific text used within a mixed-mode outlook.
At Statistics Portugal our push-to-web earlier phase communications (initial letter/email and first reminders) focus on online answering, without any mention of other modes (avoiding, for instance, mode choice burden). However, we regularly use a short reference to eventual interviewer contact (and its possible avoidance) in our email reminders preceding large scale transitions from CAWI to CATI or CAPI. The expected advantages of extending such references to an earlier stage and different contact mode are evaluated in this presentation.
Through an experimental design implemented within the 2022 Adult Education Survey data collection, we tested the impact of adding an early SMS (promoting CAWI) to our contact strategy, as well as the effect of mentioning (or not) possible contacts by interviewers.
Results will be presented regarding the timeliness and proportion of CAWI answers, as well as final response rates. Discussion of these results will also cover their implications within a segmented and tailored communication strategy in sequential mixed-mode designs, and diverse applicability in surveys with a longitudinal perspective (e.g. possible awareness of different modes after first interview).
Mrs Becky Hamlyn (Kantar Public) - Presenting Author
Ms Tugba Adali (University College London)
Dr Jake Anders (University College London)
Ms Lisa Calderwood (University College London)
Ms Xin Shao (University College London)
Mr Carl Cullinane (Sutton Trust)
Mr Keith Bolling (Kantar Public)
Mr Luke Taylor (Kantar Public)
Mr Jon Kennett (Kantar Public)
Mr Leo Brownstein (Kantar Public)
The first stage of a longitudinal survey is typically recruited face-to-face (F2F) to optimize response and allow more in-depth baseline interviews. However, suspension of F2F fieldwork during the pandemic highlighted the increased importance of push-to-web (PTW) approaches as part of mixed-mode designs.
COSMO is a new cohort study investigating the impact of COVID on longer-term outcomes of young people in England, carried out by University College London, Sutton Trust and Kantar Public, funded by UKRI/ESRC.
This paper covers the methodology of the first wave which primarily used PTW but with a F2F follow up. A representative sample was drawn from UK education records, with oversampling of harder to reach groups. The survey aimed to recruit a young person and parent at each address to complete a 30-minute questionnaire.
Conducting a PTW survey during the pandemic emphasized the need for an agile approach to survey delivery. F2F fieldwork (when allowed again) proved difficult due to interviewer panel retention and respondent reticence. This placed greater emphasis on ensuring a robust online approach. As such we made several adaptations during fieldwork: using interviewers to encourage online completion, sending out additional targeted reminders, recruiting a reserve sample, and adapting our mailing approaches to increase the rate of paired household completions.
Wave 1 recruited approximately 14,000 students and 11,000 parents. We will present a case study of the COSMO methodology including response rates, and an evaluation of representation assessed via sample frame characteristics. Wave 2 followed in 2022-23, also using an online-first approach. This achieved very high recontact rates which has helped to validate the success of the Wave 1 design. We hope this paper will provide wider lessons about optimal approaches for PTW data collection, particularly for surveys of young people.