All time references are in CEST
Push-to-Web Surveys: Challenges and Opportunities 1 |
|
| Session Organisers | Dr Olga Maslovskaya (University of Southampton) Dr Peter Lugtig (Utrecht University) |
| Time | Wednesday 19 July, 11:00 - 12:30 |
| Room | U6-08 |
We live in a digital age with widespread use of technologies in everyday life. Technologies change very rapidly and affect all aspects of life, including surveys and their designs. Online data collection is now common in many countries and some surveys started employing push-to-web approach (for example, Generations and Gender Surveys in some countries, including the UK) in which offline contact modes are used to encourage sample members to go online and complete a web questionnaire. This method of data collection is typically used when sampling frames do not include email addresses for all members of the target population. It is important to address different methodological challenges and opportunities which are associated with push-to-web surveys.
This session welcomes submissions of papers on different methodological issues associated with pish-to-web surveys in both cross-sectional and longitudinal contexts. Examples of some topics of interest are:
• Coverage issues
• Data quality issues
• Unit nonresponse
• Response rates and nonresponse bias
• Mobile device use
• Questionnaire length
• Other topics
We encourage papers from researchers with a variety of backgrounds and across different sectors, including academia, national statistics institutes, data collection organisations, and research agencies.
This session aims to foster discussion, knowledge exchange and shared learning among researchers and methodologists around issues related to push-to-web surveys. The format of the session will be designed to encourage interaction and discussion between the presenters and audience.
Keywords: online surveys, push-to-web surveys, nonresponse bias, response rates, data quality
Ms Angela Jaeger (Fraunhofer Institute for Systems and Innovation Research ISI) - Presenting Author
Ms Spomenka Maloca (Fraunhofer Institute for Systems and Innovation Research ISI)
Surveying companies is a major challenge, especially when addressing production processes or location-related information. The German Manufacturing Survey GMS 2022 was conducted online for the first time, testing different mixed-mode approaches.
Since 1995, the GMS has been one of the few surveys on production-related activities in manu-facturing. The multi-topic survey covers organisational, and technical activities as well as strategic issues such as personnel, business model or relocation and covers production sites with at least 20 employees. The respondents are the production management or the technical management. Since 2006, this survey has been addressed to enterprises in all manufacturing sectors every three years.
By means of a written survey, a representative sample was realized in each round with regard to company size, industry and regional location of the sites. Since 2012, companies have been provided with a digital copy of the survey in the form of a digital PDF to initiate the switch to a digital mode.
In 2022, the transition to an online survey took place. In a push-to-web approach, the survey was conducted in summer and successfully completed in autumn 2022. Thereby, various contact procedures were tested using a split sample design. Purely mail-based approaches, support through phone calls and email invitations were used.
The intended number of questionnaires was achieved. In addition, detailed information on the course of the survey is available for the different sub-groups. We will present analyses of the dif-ferent modes and sequences of addressing and recall. Of interest are coverage issues, the unit non-response depending of the different mixed-modes regarding company size and sub-sectors as well as the significance of different address origins. The mail returns also allow a comparative analysis of the response behaviour and the type of sites reached.
Dr Z. Tuba Suzer-Gurtekin (University of Michigan) - Presenting Author
Dr Joanne Hsu (University of Michigan)
Mrs Zhen Sun (University of Michigan)
Researchers intensely investigate properties of alternative numerical scale question designs that optimize precision required with respondent burden when they switch from computer assisted interviewing (CAI) to push-to-web surveys. A widely accepted practice in CAI is starting a question with asking a numeric response and recording a range response upon detecting uncertainty or/and unwillingness from the respondent. Long-standing Surveys of Consumers (SoC) are monthly national cell phone surveys that measure nationwide consumer expectations including inflation expectations in the U.S. Between November 2003 and November 2019, on the average 20% of respondents reported a range when inflation expectation was inquired. In parallel to phone surveys, we conducted monthly push-to-web surveys as part of SoC in the recent years. While the current adaptation of the web surveys do not allow respondents to record their responses in ranges, the percentage of don’t know responses in web surveys is higher than the percentage in phone surveys on the average. Furthermore, the difference in percentage of don’t know responses in web surveys vary by low vs. high inflation periods. In this paper, we implement a randomized experiment in the push-to-web surveys that allow respondents to enter a range response when they report their inflation expectation. We will investigate this design’s impact on break-offs, percentage of don’t knows, and response distribution statistics by experiment condition.
Dr Dina Neiger (The Social Research Centre)
Dr Benjamin Phillips (The Social Research Centre) - Presenting Author
Professor Darren Pennay (ANU)
Mr Sam Slamowicz (The Social Research Centre)
The Australian Comparative Study of Survey Methods (ACSSM) fielded in December 2022 systematically trialled a number of sampling frames and survey modes, including SMS push to web approach using Random Digit Dialling (RDD) mobile sampling frame. As well as SMS push to web (n=600), the ACSSM included Life in Australia™ probability online panel sample (n=600), two cell phone RDD CATI streams (high effort, n = 500; low effort with aggressive auto dialler settings and answering machine detection, n = 500) and four nonprobability access panels (n = 850 per panel). The study questionnaire used items for which high-quality benchmarks were available across a range of domains.
We compare the performance of SMS push to web to the other survey methods trialled with respect to response and demographic characteristics. In addition, and to the best of our knowledge for the first time, we compare SMS push to web to the external high-quality benchmarks across a range of demographic, attitudinal and health and socio-economic items. The broad range of modes included in the study help to situate performance of this sample with respect to both self- and interviewer-administered modes and probability and nonprobability frames.
We will also draw on the SRC experience in the use of SMS push to web for multiple rounds of top up and expansion of Life in Australia™ to help illustrate how performance of this method unfolded over time.
Mr Joel Williams (Kantar Public) - Presenting Author
The UK’s most longstanding push-to-web general population survey is the Community Life Survey, which was first carried out in 2012. Multiple design trials have been implemented over the last ten years: one of the most critical was to test the impact on response rates, mode ‘selection’ and respondent profile of including a paper version of the questionnaire in a reminder letter.
When the Community Life Survey was launched in 2012, the offline share of the UK population was considerably greater than it is now. Given its status as a source of Official Statistics, the Community Life Survey had to have an offline mode available for those who needed it. By the time of the trial (2020-21), the offline share of the population had fallen but perhaps become more distinctive. It was certainly time to review how the paper questionnaire was used. In particular, both the clients and the Kantar Public research team were concerned that lower quality paper data was displacing higher quality web data when the choice was offered to sampled individuals.
In this paper, we review the results from the 2020-21 randomised controlled trial to identify the additive value of proactively providing paper questionnaires as opposed to providing them on request. In particular, we evaluate the evidence for the ‘displacement’ theory outlined above and discuss the consequent implications for the design of this survey and perhaps others that employ similar methods.
Miss Grace Chang (University of Southampton) - Presenting Author
Dr Olga Maslovskaya (University of Southampton)
Professor Brienna Perelli-Harris (University of Southampton)
Using the first ever Generations and Gender (GGS) survey conducted in the UK, this study examines the representativeness of the survey in relation to the UK population, based on the UK Census 2022. The UK GGS is a push-to-web survey that uses a stratified random probability selection of addresses in the UK through postcodes. One of the challenges in the UK for online data collection is the absence of an individual-level sampling frame. We examine whether the UK GGS is representative of the 18 – 59 young adult population in the UK by gender, ethnicity, deprivation and fertility, among other demographic characteristics. Preliminary results for stage 1 of data collection has been conducted recently, but final results will be available by May 2023 when all data are collected for GGS in the UK. UK specific weights will be calculated and will also be used for this analysis. This study seeks to discuss the strengths and challenges with using a push-to-web survey design in the UK, which will provide useful insights for other countries which do not have population registers and desire to move to push-to-web data collection.