Generating Methodological Insights from the ESS Probability Based Online Web Panel CRONOS 2 |
|
| Session Organisers |
Dr Gianmaria Bottoni (European Social Survey, City University of London ) Professor Rory Fitzgerald (European Social Survey, City University of London ) Dr Elena Sommer (SHARE ERIC) Dr Ana Villar (Facebook) |
| Time | Wednesday 17th July, 09:00 - 10:30 |
| Room | D26 |
This session is open to anyone who would like to analyse data and experiments from the CROss-National Online Survey panel (http://bit.do/CRONOS_DATA), a project led by the European Social Survey, to explore the feasibility of using existing survey infrastructures to recruit probability-based samples for online panels. The CRONOS project capitalised on an existing probability-based face-to-face survey to establish a probability-based sample for an online panel which now comprises data from seven waves, together with experimental data and paradata. All this data is available to any researcher interested in cross-national survey research methodology and it can be downloaded for free from the ESS website.
CRONOS was part of the Horizon 2020 grant Synergies for Europe’s Research Infrastructures in the Social Sciences (SERISS). The panel followed an input-harmonisation approach, with the survey design coordinated by ESS ERIC with crucial continuous support from national teams and other SERISS partners. Using these harmonised protocols, the national teams in Estonia, Great Britain and Slovenia carried out the recruitment of panel members on the back of the European Social Survey (ESS). All ESS sampled units 18 or older were eligible to participate in the panel; after completing the ESS Round 8 (2016) face-to-face interview. Those who agreed were invited to participate in six 20-minute online surveys over a period of 12 months.
Apart from a large amount of substantive data, the CRONOS panel served as a platform for methodological testing. Across the six CRONOS waves, numerous methodological and substantive areas were covered. Methodological efforts included pretesting of new questions, experiments on question wording and satisficing, incentive approaches, contact modes, and other studies. Substantive questions covered various topics related to family, religion and values amongst others from established cross-national studies such as the Generations and Gender Programme (GGP), the European Values Study (EVS) and the International Social Survey Programme (ISSP). The CRONOS data can be analysed together with responses from the ESS face-to-face interview, allowing for new substantive and methodological analyses which was not previously possible.
This session invites papers on methodological findings from the panel. Papers might cover: representativeness, cost analyses, contact mode effects and incentive strategies, effects of device on measurement, efforts to improve survey completion respondent behaviour and the impact of including off-liners through device provision.
Keywords: CRONOS, ESS, online survey panel, methodological experiments, mode effects
Mr Indrek Soidla (University of Tartu) - Presenting Author
Mrs Mare Ainsaar (University of Tartu)
Mrs Ana Villar (Facebook)
Mr Slavko Kurdija (University of Ljubljana)
Ms Tina Vovk (University of Ljubljana)
Mr Alun Humphrey (NatCen Social Research)
Social stratification studies have rarely been based on data from web surveys or web panels because of methodological challenges related to representativeness of data. Web surveys tend to underrepresent groups with lower social status and lower educational level because of differences in internet access and online skills, and probably other reasons. As regards attrition, evidence from longitudinal studies in other survey modes shows higher dropout rates among the lower-educated, while other studies have found that attrition by income levels follows a U-shape.
The aim of the presentation is to analyse the representativeness and attrition in the CRONOS web panel in terms of social stratification as measured by social status, education and income variables. The CRONOS web panel was carried out in the United Kingdom, Slovenia and Estonia within the ESS-led SERISS project to investigate the feasibility of conducting cross-national probability-based web panels. The combination of CRONOS and ESS Round 8 data along with information about the survey design allows us to investigate the role of various individual and household-level characteristics related to socioeconomic status in attrition. The combination of ESS8 and CRONOS data also enables us to find the impact of these factors on data quality in different waves and analyse it in combination with different attrition types (early/late, complete/partial attriters) during the web panel. Preliminary results show that attrition over waves is higher in lower-educated groups in all three countries, whereas attrition by occupational strata and household income levels is more mixed.
Dr Gianmaria Bottoni (European Social Survey (ESS) ERIC HQ, City, University of London) - Presenting Author
Professor Rory Fitzgerald (European Social Survey (ESS) ERIC HQ, City, University of London)
Online surveys have gained substantial popularity amongst academic scholars and for commercial use. The increased use of web surveys is related to several advantages – lower costs, shorter setting up time and more flexibility in terms of research design. There are also shortcomings with online panels. Besides non-coverage and self-selection bias related to so-called volunteer panels, response rate has been a concern with many web surveys. In addition, there is the risk of a constant confusion between non-scientific and scientific web surveys (Couper 2000). This can further lower response rates.
CRONOS represents the world’s first academically driven cross-national probability-based input harmonised web panel. The same methodological procedures – implementation, recruitment, setup and maintenance of the panel – have been implemented in all the countries joining the project to reflect this input harmonised framework.
Aiming at testing the feasibility of implementing a cross-national probability-based online panel, CRONOS represented also a proof of concept useful to set up several methodological experiments.
In this study, we assess the effect of varying the contact mode on the cooperation rate analysing an experimental design with three conditions: SMS pre-notification, e-mail pre-notification and no pre-notification. The aim is to identify the best contact mode strategy to increase survey participation. In addition, the paper also evaluates whether the effect is similar across countries or not. Finally we try to identify panellists’ characteristics that are related to preferences toward a specific pre-notification mode.
Dr Elena Sommer (Survey of Health, Ageing and Retirement in Europe (SHARE), Munich Center for the Economics of Aging (MEA)) - Presenting Author
Dr Ana Villar (Facebook)
Research has consistently demonstrated that unconditional incentives is an effective strategy to increase survey participation affecting how respondents react to survey invitations by, among others, improving trust in the researchers and creating a sense of duty. In online panel surveys that are conducted on frequent basis, providing unconditional incentives implies a strategic decision on incentives type. In our contribution, we first evaluate the effectiveness of online incentives (Amazon vouchers) and postal incentives (gift-cards) using data from the Great Britain sample of the probability-based CROss-National Online Survey (CRONOS) that was conducted bimonthly between December 2016 and February 2018. The postal gift-card approach implemented in Wave 1 has resulted in higher participation than the online voucher that was used in the Welcome wave. An experiment to compare two approaches to postal incentive delivery was implemented in Wave 1 – Wave 6: 1) a wave-by-wave approach, where panellists received a £5 postal gift-card with every wave pre-notification and 2) a one-off approach, where panellists received the full incentive amount £30 for 6 waves upfront with the first pre-notification. The latter approach lowers administration cost but is associated with a risk of potential higher attrition in later waves. As expected, the £30 gift-card of the one-off approach led to higher participation in Wave 1 as compared to the £5 gift-card of the wave-by-wave approach. Participation in subsequent waves was similar in both groups. The two groups, however, differed in their self-reported retrospective survey participation motivation. Whereas the £5 group was more likely to join the panel in order “to help scientific research” and reported higher “trust in the conclusions that researchers will make from this study”, the £30 group was more likely to be guided by a sense of duty staying in the panel because they “made a commitment".
Mr Oriol J. Bosch (Research and Expertise Centre for Survey Methodology (RECSM)-Universitat Pompeu Fabra) - Presenting Author
Dr Wiebke Weber (Research and Expertise Centre for Survey Methodology (RECSM)-Universitat Pompeu Fabra)
Dr Melanie Revilla (Research and Expertise Centre for Survey Methodology (RECSM)-Universitat Pompeu Fabra)
Respondent motivation during the survey response process can differ across individuals and situations, leading to suboptimal responding behaviours. Survey methodologists have been searching for strategies to identify and reduce this kind of behaviours especially among online respondents. Motivating respondents might increase the number of answers and the quality of these. However, if these answers are given without appropriate thinking, this can lead to lower data quality. Past research found little evidence of an association between displaying motivational messages and a reduction of break-offs or an increase of data quality. However, past research is mainly based on studies within a single country and on opt-in panels. More research is needed.
This paper presents the results of an experiment conducted in waves 2, 4 and 6, between April 2017 and February 2018, of the CROss- National Online Survey (CRONOS) panel in Estonia, Slovenia and Great Britain. Respondents were randomly assigned to a control group and two treatment groups (message emphasizing the positive or the negative consequences of their behaviours) in wave 2 and were kept in the same group for waves 4 and 6. Four or five other motivational messages (depending on wave) were placed throughout the questionnaire. In order to establish the impact of the motivational messages, we compared control and treatment groups in terms of break-offs, item nonresponse, non-differentiation, completion time, efforts reported and evaluation of the survey experience.
We found little impact of motivational messages on the different aspects. Moreover, positive and negative messages have different impacts depending on the country, suggesting that strategies to motivate respondents should be country-specific. Besides, significant impacts on data quality were mainly found on the first questions early in the questionnaire. Thus, the cumulative effect decreases across the questionnaire, i.e. respondents being tired of repeatedly being exposed to motivational messages.