All time references are in CEST
Cross-national probability-based web or mixed mode panels 3 |
|
| Session Organisers | Dr Gianmaria Bottoni (ESS ERIC HQ, City University of London) Professor Rory Fitzgerald (ESS ERIC HQ, City University of London) |
| Time | Friday 21 July, 09:00 - 10:30 |
| Room | U6-09 |
Conducting high quality scientific social surveys in a national setting is per se a complex, costly and time-consuming task, but cross-national surveys face significant additional challenges. The historical development of the social survey reflects the ongoing tension between trying to maintain or reduce costs whilst also trying producing high quality data. The development of Information and Communication Technology has opened a new chapter in that complex interrelationship. Web surveys have several characteristics that make them an attractive alternative to more traditional modes of data collection and in particular to face-to-face surveys which are becoming increasingly expensive. With web surveys there are no costs of paying and training interviewers and the interviewer effect is removed from the equation. Answers to the questionnaire are captured in real time and reminders can be sent in a digital format once respondents are identified. In addition, the gap between questionnaire design and the start of fieldworks is substantially reduced.
For these reasons, web surveys are particularly suitable for implementing panel studies as once willing respondents are recruited their responses can be repeatedly captured without further in-person contact.
In the last years, several countries in Europe, Australia and USA have established on-line or mixed mode panels with a web component based upon probability samples.
This session is open to anyone who would like to present methodological findings from existing or planned cross-national web panels. Amongst the other topics, papers might cover: challenges linked to cross-national setting (e.g. input-harmonisation, translation issues, questionnaire design) representativeness, recruitment strategy, cost analyses, contact mode effects, incentive strategies, effects of device on measurement, mode effect, efforts to improve survey completion respondent behaviour, strategies to improve response rate, and the impact of including off-liners through alternative modes of data collection.
Keywords: Online survey methods, Cross-national web panels
Dr Peter Grand (Institute for Advanced Studies (IHS)) - Presenting Author
Survey research faces considerable obstacles while simultaneously technological advances offer new possibilities. F2F-Interviews became more and more expensive and response rates are lowering in a wide range of countries. During the worldwide COVID19 pandemic it became nearly impossible to conduct F2F-interviews and even if they have been possible, we do not know what consequences this context has had on the data quality. Thus, big survey projects are looking for new ways of data collection which are in line with the pre-requisites of statistical sampling theory in order to uphold high methodological standards. ESS ERIC conducted the first experiments using a push-to-web self-completion approach for a possible future mode change. This paper compares the Austrian data of this experiment to the Austrian data of ESS round 9 and the respective official statistical data. Since respondents could choose between conducting the survey via web or - received together with the last reminder - using a paper questionnaire this paper also assesses differences between these two modes. In Austria, Hungary and Serbia the ESS conducted a push-to-web self-completion experiments using a web survey and paper questionnaire. In this paper I will present the Austrian data, compare it to external data sources and analyse possible mode effects using ESS round 9 data from Austria. I follow different strategies to assess possible mode effects, comparison to official statistics, comparisons of means and standard deviations between the ptw and ESS9 data, comparing empirical cumulative distributions between those two datasets and assessing satisficing. This push-to-web experiment has been the first try to find a suitable solution within the ESS survey infrastructure to switch the mode of data collection. This paper compares (1) the push-to-web data with validated official statistics, (2) ptw and ESS 9 data using means and standard deviations, (3) ptw and ESS 9 data assessing the respective empirical cumulative distributions for each response option, and, (4) finally, assessing satisificing comparing the two modes in the ptw data and the ptw data to ESS 9 data using a so-called differentiation index. This Push-to-web experiment has been the first try to find a suitable solution within the ESS survey infrastructure to switch the mode of data collection. The empirical investigation so far yielded promising results that a switch to a self-completion push-to-web mode is a viable strategy to cope with most of the problems the “classical” F2F-approach is plagued with. The sample composition of the PtW data is pretty in line with the ESS 9 data except the overrepresentation of older cohorts which may vanish as the “new” technologies become a common tool for all age groups. Age is still a significant driver of selection effects, i.e. as a predictor for either using a web-based questionnaire or a paper questionnaire, in a self-completion mode.
Dr Vera Messing (Center for Social Sciences) - Presenting Author
Dr Bence Ságvári (Center for Social Sciences, Corvinus University of Budapest)
Mr Ádám Stefkovics (Center for Social Sciences)
Ms Blanka Szeitl (Center for Social Sciences)
The proposed presentation will highlight the challenges and solutions in recruiting online web panels through probability-based face-to-face and push-to-web surveys in Hungary and compare panel composition using two different survey modes for recruitment. As part of the ESS SUSTAIN-2 project a webpanel was recruited through the interviewer-assisted face-to-face survey of ESS R10 in 12 countries in 2021/22. Recruitment rates proved poor and the effective sample size achieved was unsuitable for further analysis in Hungary. To increase the size of the webpanel (CRONOS-2), the Hungarian team launched a probability-based mixed-mode self-completion survey (push-to-web design), inviting respondents by post to go online or complete a questionnaire. The questionnaire was identical to the interviewer-assisted ESS R10 survey. Recruitment via the self-completion questionnaire resulted in a significantly higher number of recruited panel members compared to the original interviewer-assisted survey. In our presentation, we will first introduce the design of the two surveys, then explain the challenges in setting up the panel, and finally compare the composition of the panel recruited through the two surveys (interviewer-assisted ESS R10 and push-to-web survey with self-completion). Our research offers important insights into how recruitment to a web panel through probability sampling surveys is influenced by the type of survey. However, the results cannot be automatically generalised to other countries, as we suspect that the socio-political context and the level of trust have an important influence on the willingness to participate in different survey modes.
Mr Carsten Broich (Lifepanel)
Ms Elena Babamova Bozhinovska (Lifepanel)
Ms Nadica Stankovikj (Sample Solutions)
Mr Nikola Jovanovski (Lifepanel) - Presenting Author
With the start of the pandemic, European survey research has witnessed a shift between offline sampling frames: Face-to-Face (F2F) migrating to telephone Random-Digit-Dialing (RDD) as evident with the European Working Conditions Survey(EWCS) by Eurofound and Gallup’s World Poll.
Another shift of survey modes includes migrating from offline online survey modes however, as there is not definitive online sample frame of entire populations. Online surveying is leveraged though building probability based via recruitment from offline sampling frames such as ABS (via F2F and postal invitations), RDD (telephone) or a combination of both.
The selection of the offline frames for panel recruitment depend on a country-per-country basis, due to differences in sample coverages, frame sizes and availability of population registers.
After selecting the approproate sampling frame for the specific country, there is still the question on how to execute the recruitment of the chosen interview mode.
Researchers are tempted to conduct the panel recruitment upon finishing a questionnaire response - thus leveraging a “Piggyback” approach of already engaged respondents.
While at the other hand, researchers might consider using a “stand-alone” recruitment where respondents are given a general introduction of the organisation's background, topics for research and early mention of incentives.
Using the recent case of Lifepanel, which applied the two strategies in a telephone recruitment with the utilisation of Dualframe RDD sampling for Germany, this paper outlines side by side comparisons of the efficiencies in recruitment rates, listing what would be the expected mode effects and drawbacks of either strategy as well as suggesting future considerations on how to improve both approaches.
Dr Joke Depraetere (Ipsos EPA)
Dr Femke De Keulenaer (Ipsos EPA) - Presenting Author
Ms Cristina Tudose (Ipsos EPA)
Ms Christine Tresignie (Ipsos EPA)
In today's digital age and high demands for speed and cost-effectiveness, traditional random probability methods are under pressure. These methods are often associated with high costs, are time-consuming, and show increasing difficulty in reaching hard-to-reach subgroups (low income, low education, youth, minorities, offline population). While the acceptance of online access panels came about because of the need for faster turnaround time, the struggle to publish results that can withstand academic scrutiny remains. KnowledgePanel covers many of these aspects and differentiates from traditional opt-in online panels by its underlying methodology. KnowledgePanel is a random-probability online panel and has been operating in the US since 1999 and in the UK since 2020. Building on this work, we are now expanding the offering across Europe.
Although probability-based recruitment is considered the gold standard in research, it is not without its challenges. Recruiting hard-to-reach populations (e.g. youth/65+, low-educated, those not internet-savvy), while maintaining a random probability strategy, remains an ongoing challenge, as well as panel engagement and attrition. An additional challenge for KnowledgePanel in Europe is related to its cross-national design. While the cross-national set-up makes it possible to unify the approach, differences between countries - such as local framework, available sampling resources and constraints - must also be taken into account. Solutions that work in some countries are not feasible in others, emphasizing the need for adopting a more flexible approach.
In this presentation, we highlight the methodological, practical and cross-national challenges faced during the KnowledgePanel recruitment phase. Results of several experiments and pilot test focusing on improving the recruitment rate and panel engagement are presented. With this presentation, we focus on lessons learned from the launch of KnowledgePanel Europe and provide a basis for future research.
Mr Samuel Slamowicz (The Social Research Centre) - Presenting Author
Dr Benjamin Phillips (The Social Research Centre)
Dr Dina Neiger (The Social Research Centre)
Professor Darren Pennay (Australian National University)
The Australian Comparative Study of Survey Methods (ACSSM) fielded in December 2022 systematically trialled nine sampling frames and survey modes: web mode using a probability-based online panel, Life in Australia™ (n = 600), ™, a video-assisted live interviewing arm recruited from Life in Australia™ (n = 600), two mobile phone RDD CATI streams (high effort, n = 500; low effort with aggressive auto dialler settings and answering machine detection, n = 500), RDD SMS push-to-web (n = 600), and four nonprobability access panels (n = 850 per panel). The ACSSM follows an earlier Australian comparative study (Lavrakas et al. 2022) fielded in 2015, which compared various probability and nonprobability modes, and its replication on Life in Australia™ in 2017.
The questionnaire used items for which high-quality benchmarks were available across a range of domains including health, substance use, disability, caring, psychological distress, labour force status and voting. We deliberately included items likely subject to mode effects.
We compare relative strengths and weaknesses of these methods and compare the performance of the trialled survey methods with respect to the external high-quality benchmarks. Being in part a repetition of an earlier study, we are able to examine how performance of methods changed over time. This paper contributes to the multinational evidence base on the performance of contemporary and emerging methods for general population surveys.