All time references are in CEST
Cross-national probability-based web or mixed mode panels 1 |
|
| Session Organisers |
Dr Gianmaria Bottoni (ESS ERIC HQ, City University of London) Professor Rory Fitzgerald (ESS ERIC HQ, City University of London) |
| Time | Thursday 20 July, 14:00 - 15:30 |
| Room | U6-01b |
Conducting high quality scientific social surveys in a national setting is per se a complex, costly and time-consuming task, but cross-national surveys face significant additional challenges. The historical development of the social survey reflects the ongoing tension between trying to maintain or reduce costs whilst also trying producing high quality data. The development of Information and Communication Technology has opened a new chapter in that complex interrelationship. Web surveys have several characteristics that make them an attractive alternative to more traditional modes of data collection and in particular to face-to-face surveys which are becoming increasingly expensive. With web surveys there are no costs of paying and training interviewers and the interviewer effect is removed from the equation. Answers to the questionnaire are captured in real time and reminders can be sent in a digital format once respondents are identified. In addition, the gap between questionnaire design and the start of fieldworks is substantially reduced.
For these reasons, web surveys are particularly suitable for implementing panel studies as once willing respondents are recruited their responses can be repeatedly captured without further in-person contact.
In the last years, several countries in Europe, Australia and USA have established on-line or mixed mode panels with a web component based upon probability samples.
This session is open to anyone who would like to present methodological findings from existing or planned cross-national web panels. Amongst the other topics, papers might cover: challenges linked to cross-national setting (e.g. input-harmonisation, translation issues, questionnaire design) representativeness, recruitment strategy, cost analyses, contact mode effects, incentive strategies, effects of device on measurement, mode effect, efforts to improve survey completion respondent behaviour, strategies to improve response rate, and the impact of including off-liners through alternative modes of data collection.
Keywords: Online survey methods, Cross-national web panels
Dr Gianmaria Bottoni (ESS HQ, City, University of London) - Presenting Author
Professor Rory Fitzgerald (ESS HQ, City, University of London)
Several countries in Europe and in other parts of the world have established successful, high quality national web panels based on probability samples. However, most of them are focused on country-specific aims, not easily allowing for reliable comparative analysis. With the exception of CRONOS (the world’s first cross-national probability-based web panel), there are still few probability based, input harmonised cross-national panels that allow for a comparative analysis of the social condition and attitudes of different national populations.
Based upon the previous experience with CRONOS, the European Social Survey ERIC has tried to establish a cross-national probability-based web panel on a larger scale, this time involving 12 European countries (Austria, Belgium, Czechia, Finland, France, Hungary, Iceland, Italy, Portugal, Slovenia, Sweden, and UK). The paper will describe the process of building the panel including the methods used and the technology utilised (such as the Web Panel Sample Service). The paper will then discuss the main challenges and results of the project focusing on the cross-national differences in the recruitment and response rates from the participating countries.
Dr Noura Insolera (University of Michigan, Institute for Social Research) - Presenting Author
Dr Katherine McGonagle (University of Michigan, Institute for Social Research)
The Panel Study of Income Dynamics (PSID) is a longitudinal study of a nationally representative sample of U.S. families. Begun in 1968, the study was fielded annually until 1997, and biennially thereafter. Since the beginning of the study, families have been offered a modest incentive in exchange for returning a postcard via U.S. postal mail on which they verify or update their contact information (addresses, phone numbers, email addresses) with the goal of keeping track of their whereabouts between waves of data collection. Having updated contact information prior to data collection is associated with lower fieldwork effort and refusal conversion efforts, less tracking, and lower attrition. As part of the transition in 2021 to the use of mixed modes to collect survey data in PSID, we also redesigned the between-wave mail-based contact strategy used for many decades to include multiple modes (mail/web) with the goals of improving participation and reducing costs. In advance of the 2023 wave, over 11,500 families were sent a mailing and could choose to either return the information via postal mail or provide an update on the web. To respond by web, respondents were instructed to scan a quick response (QR) code printed on the postcard that took them to the contact information web instrument. These QR codes contained secure, authenticated links to a prefilled contact information listing for their family. To date, the mixed mode response rate exceeds the prior wave mail-based approach. This presentation describes the mixed mode contact update design and protocol, its response rate, costs, and socio-demographic characteristics of participating families compared to the traditional mailing used in the prior wave.
Dr Jaroslava Pospíšilová (Institute of Sociology of the Czech Academy of Sciences) - Presenting Author
In the Czech Republic, there are currently several online panels operated by agencies mainly for marketing research, recruitment to the panels takes place through various channels, and the selection of respondents for participation in a specific study is subsequently carried out on a quota basis. As part of the CRONOS-2 project, we created an entirely new panel based on a probabilistic selection of ESS Round 10 respondents, using a piggy-backing approach. Since it is impossible to use population registers for research design in the Czech Republic, a random probability sample was created using an address-based sampling frame, a multistage sampling design with systematic sampling at the first two stages, and a random respondent selection at the last stage. Although the recruitment strategy based on the parent study was associated with the expectation of a higher response rate, the final number of participants was rather disappointing. During the face-to-face recruitment, approx. 30% of respondents to the ESS Round 10 agreed with participating in the online panel; however, only 25% subsequently participated in the welcome survey. Some additional actions were taken to increase the number of participants; however, most of them were with little success. The total participation rate stayed far below expectations. As previous studies show, a low response rate is not necessarily the source of low data quality. As another alternative metric, some authors propose the representativeness of the resulting sample. In the proposed paper, we will try to find the main reasons for low participation and name the main pitfalls of creating a panel. We will discuss the panel recruitment process in detail, look at the characteristics of the resulting sample of panelists, analyze its representativeness and compare the characteristics of active and inactive ones, and evaluate the the effectiveness of follow-up activities.
Dr Michael Bergmann (Technical University of Munich )
Dr Yuri Pettinicchi (SHARE Berlin Institute) - Presenting Author
Dr Elena Sommer (SHARE Berlin Institute)
There is a tendency among European cross-national probability-based surveys to move to the web mode or to a mixed-mode survey design. The Survey of Health, Ageing and Retirement in Europe (SHARE) intends to keep face-to-face interviews as the main mode of data collection at least in the near future but there are plans to introduce other modes of data collection, including web surveys, as a complementary mean to collect additional timely data in-between main face-to-face waves. There are concerns whether web is a feasible mode of data collection for the SHARE target population of respondents 50+ as there is a large proportion of (older) respondents not using the internet which is additionally varying across participating countries. So far, there are no methodological studies on participation in web surveys targeting explicitly at the older European population and using probability-based samples in a harmonized cross-national context.
As part of SHARE’s preparation for its multi-mode panel future, it is planned to conduct an experiment, which aims at evaluating the feasibility of the web mode for SHARE. At the end of the SHARE Wave 10 (planned for 2024) face-to-face interview, respondents in selected countries who stated that they use the internet will be invited to participate in a short self-completion survey. They will be randomly assigned to receive either a mailed paper questionnaire or an invitation to an online survey. The content of the surveys will be the same. In the second wave of the experiment, respondents will be randomly assigned to four experimental groups, either keeping or switching the respective self-completion survey mode. We present the design plans of this experiment and focus on the challenges for the technical infrastructure set-up and the cross-national implementation.