All time references are in CEST
Tailoring push-to-web for different audiences |
|
| Session Organiser |
Ms Eileen Irvin (Ipsos) |
| Time | Wednesday 19 July, 09:00 - 10:30 |
| Room | U6-01a |
In the last few years, there has been a massive increase in push-to-web surveys being used as an alternative to traditional paper and face-to-face surveys. This is due to a combination of factors, including shrinking budgets, increasing acceptability of online methods, and the challenges of face-to-face fieldwork during the COVID-19 pandemic.
However, as this method becomes more widely used, it is important to think not just about what works for 'general populations', but also how to tailor this method to meet the needs of everyone. This could be for an entire survey with a particular population, or for a sub-group within a general population survey.
This tailoring has two key implications:
1. Firstly, meeting any ethical or safeguarding concerns for populations. For example, this could include ensuring that the speed, number and nature of contacts do not agitate those living with anxiety disorders for surveys targeted at people living with mental health conditions (e.g. as on CQC's Community Mental Health survey).
2. Secondly, reducing non-response bias. Ensuring populations who may be less likely to respond to traditional push-to-web strategies are approached using a method that works for them can reduce non-response bias (e.g. on the Ethnic Minority British Election Study (EMBES)).
Tailoring of approach could include designs of materials, targeted incentives, standardised inclusion of translated or accessible materials, or reviewing contact strategies.
Understanding what is happening to tailor push-to-web, for which populations, and in what ways, will be useful to expand how this method is used in the future.
Keywords: push-to-web, web-push, online, tailoring, non-response bias,
Ms Alexandra Asimov (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Mr Michael Blohm (GESIS - Leibniz Institute for the Social Sciences)
Push-to-web is a common strategy for self-administered mixed-mode surveys as it reduces survey costs and web surveys can lead to better data quality than mail surveys, for example, due to automatic routing. However, in general social surveys, push-to-web can lead to lower representativeness than a concurrent design. To compensate for this problem, some surveys have tailored the mode choice sequences. For this purpose, an indicator is used to decide which respondents are unlikely to participate in the web mode and those individuals are switched directly to concurrent design. The remainder of the sample follows the push-to-web design. Some general social surveys, such as the European Values Study in Germany, use the age as an indicator. One reason for this is that when population registry is used for sampling, age of the sampled person is known before the survey. Another reason is that older people are less willing to participate in an online survey. However, there is a lack of research on the optimal cutoff point of age. We implemented a randomized experiment in the German General Social Survey 2021 (‘ALLBUS’, which surveys the German general population aged 18 and older; N=5,342), which was conducted in mixed-mode design (web and mail). Respondents were randomly assigned to one of the two mode choice sequences: concurrent and push-to-web. To determine whether age is a good indicator for mode choice sequences, we examine whether and at what age a switch to a concurrent design reduces nonresponse bias. First, we examine at what age the response rate decreases in the push-to-web design. Next, we use this value of age as a cutoff point in the push-to-web, from which we simulate a concurrent design to examine how representativeness changes.
Ms Aneta Malina (infas Institute for Applied Social Sciences) - Presenting Author
Dr Merih Ates (German Center for Integration and Migration Research)
Mr Moritz Fahrenholz (infas Institute for Applied Social Sciences)
Dr Mujtaba Isani (German Center for Integration and Migration Research)
Dr Sebastian Link (infas Institute for Applied Social Sciences)
Mr Michael Ruland (infas Institute for Applied Social Sciences)
Ms Lisa Walter (German Center for Integration and Migration Research)
In recent years, there has been a trend in empirical social research towards "computer-assisted web interviews" (CAWI) and the increased use of online surveys (push-to-web). This paper focuses on the use, challenges and opportunities associated with online surveys of migrant populations.
Based on the results of the initial survey of the NaDiRa.panel, which was conducted in 2022 as a PAPI and CAWI study, we examine differences in the use of the survey method between different migrant groups. In the context of this study, persons of the German living population with and without a migration background were surveyed. The study included an oversample of four immigrant background groups in Germany: people from Turkey, other majority Muslim countries, East- and Southeast Asia and countries of west, south, east and central Africa. The NaDiRa.panel offers some unique possibilities to examine difficulties and chances when it comes to pushing different and sometimes very small migrant populations to web surveying.
All participants had the opportunity to take part in the survey both online (CAWI) and paper-based (PAPI). For easier access to the online questionnaire (which was accessible in six different languages), a personalised QR code was printed on the cover letter and the questionnaire in addition to the access data in the cover letter (link and personalised password).
In this paper we highlight and compare results on the use and selection of the CAWI method in the different and sometimes hard-to-reach population groups. First results show that the use of CAWI differs strongly between the groups. These differences are not explained by compositional differences regarding socio-demographic characteristics, proficiency of the German language or the duration since immigration, but instead point to more complex group differences (e.g. specific experiences of a group).
Ms Eileen Irvin (Ipsos, UK) - Presenting Author
The Community Mental Health Survey, part of the Care Quality Commission’s (CQC) NHS Patient Survey Programme, asks about people’s experiences of NHS community mental health services in England. The programme is used by CQC, government, NHS England, and healthcare trusts for regulatory activities, service improvement, and policy design and measurement.
CQC commissioned Ipsos to transition the programme from paper-only to push-to-web, where participants receive paper invitations with online log-in details, with a paper questionnaire provided to non-respondents. This approach can be more convenient for respondents, more cost effective, and provide better data quality.
Ipsos have carried out a programme of work to support this change in mode, tailoring to the population and measuring mode effects. This involved extensive stakeholder consultation and running experimental pilots, comparing paper-only to push-to-web. As participants were receiving support for mental health conditions (the survey covers psychotic, non-psychotic, and dementia-related conditions), it was important to consider ethical implications of the approaches, to minimise burden and distress. As the survey highlights key areas for service improvements and can be used to identify health inequalities, it was vital the alternative approach continued to deliver robust findings, and reduced non-response bias.
Compared to a standard push-to-web design, the approach for the pilot adjusted the number and timing of contacts, updated materials, and reviewed accessibility and support arrangements, to ensure this approach was effective and appropriate. The design process was underpinned by engagement with stakeholders, charities, service users and carers. The pilot showed that this approach could work effectively, responding to ethical concerns and robustly collecting data from this vulnerable population. Based on the findings of the pilot, the change in mode is currently being rolled out on the mainstage survey.
Ms Samantha Clemens (Ipsos)
Mr Nicholas Gilby (Ipsos) - Presenting Author
Ms Galini Pantelidou (Ipsos)
The REACT1 (Real-time Assessment of Community Transmission) study was a large survey designed to measure the prevalence of COVID-19 in the general population (aged 5+) of England.
It aimed to achieve 100,000 – 150,000 completed PCR tests per month, and gather information about people’s health and behaviour. It ran from May 2020 to March 2022, using a push-to-web approach, with samples drawn from the NHS database of people registered with a GP. Letters were sent inviting people to register online or by phone. Those registering were sent a PCR testing kit and a survey link. PCR tests were collected by courier (or posted) and taken to a lab for processing.
Response rates varied over time from 30% to 12%. To maintain response we adapted the reminder regime and changed the messaging in the letters.
The sample frame included the gender and age of those sampled. The response patterns showed those living in deprived areas and some specific age groups (particularly 5-12, 18-34 and 85+ age groups) we under-represented.
Targeted incentives are not widely used in social surveys in the UK. However, offering universal incentives is costly and statistically inefficient. REACT1 successfully used targeted incentives on a scale unprecedented in a UK social survey, while minimising the cost. First, we experimented to assess whether using incentives for specific age groups could improve the representativeness of our achieved sample. The experiment’s results suggested that this approach could improve the sample efficiency, and we implemented a targeted incentive approach for the final two rounds of the study. This successfully improved the representativeness of the achieved sample.
We also experimented with tailored letters, with specific messaging for parents of younger children (aged 5-12) and for older respondents (aged 70+). This had mixed success.
Dr Nicole Martin (University of Manchester) - Presenting Author
Professor Maria Sobolewska (University of Manchester)
Ms Alex Bogdan (Ipsos UK)
Professor Ed Fieldhouse (University of Manchester)
Dr Jonathan Mellon (West Point United States Military Academy)
Professor Stephen Fisher (University of Oxford)
Robust research into political attitudes of people from ethnic minority backgrounds in Great Britain is lacking. Latest Census reveals a growing population of people from ethnic minority backgrounds. Yet, the largest representative study of political and social attitudes of Britain’s ethnic minority population, the 2010 Ethnic Minority British Election Study (EMBES) – a face-to-face probability study – is over a decade old.
In the context of increased challenges in conducting face-to-face research, including rising costs and difficulties associated with Covid restrictions, and building on the successful transition of BES2019/2020 to push-to-web as an immediate response to Covid, the EMBES 2021 Pilot investigates the feasibility of conducting research among this population using a push-to-web methodology. While push-to-web has been assessed to produce reasonably representative results for studies of general population, we address its suitability for recruiting a new representative sample of ethnic minorities.
Our paper will focus on response rates, which were lower than expected. We leverage the fact that we used two recruitment approaches, the second one designed to boost response rate but incidentally allowing us to study response rates of white respondents living in the targeted areas of high ethnic diversity. The second approach shows that the poor response rates were not limited specifically to ethnic minority respondents, and indicate that general population sample might under-represent geographies where minorities usually concentrate. We will review the impact of various geographical factors on response to push-to-web surveys, in direct comparison with face-to-face studies. We will use the EMBES 2021 Pilot, as well as British Election Studies (BES) 2019/2020 and BES 2017 to explore whether push-to-web systematically underrepresents high-density and deprived geographies. Our results therefore will have important lessons not just for ethnic