Generating Methodological Insights from the ESS Probability Based Online Web Panel CRONOS 3 |
|
| Session Organisers | Dr Gianmaria Bottoni (European Social Survey, City University of London ) Professor Rory Fitzgerald (European Social Survey, City University of London ) Dr Elena Sommer (SHARE ERIC) Dr Ana Villar (Facebook) |
| Time | Wednesday 17th July, 11:00 - 12:30 |
| Room | D26 |
This session is open to anyone who would like to analyse data and experiments from the CROss-National Online Survey panel (http://bit.do/CRONOS_DATA), a project led by the European Social Survey, to explore the feasibility of using existing survey infrastructures to recruit probability-based samples for online panels. The CRONOS project capitalised on an existing probability-based face-to-face survey to establish a probability-based sample for an online panel which now comprises data from seven waves, together with experimental data and paradata. All this data is available to any researcher interested in cross-national survey research methodology and it can be downloaded for free from the ESS website.
CRONOS was part of the Horizon 2020 grant Synergies for Europe’s Research Infrastructures in the Social Sciences (SERISS). The panel followed an input-harmonisation approach, with the survey design coordinated by ESS ERIC with crucial continuous support from national teams and other SERISS partners. Using these harmonised protocols, the national teams in Estonia, Great Britain and Slovenia carried out the recruitment of panel members on the back of the European Social Survey (ESS). All ESS sampled units 18 or older were eligible to participate in the panel; after completing the ESS Round 8 (2016) face-to-face interview. Those who agreed were invited to participate in six 20-minute online surveys over a period of 12 months.
Apart from a large amount of substantive data, the CRONOS panel served as a platform for methodological testing. Across the six CRONOS waves, numerous methodological and substantive areas were covered. Methodological efforts included pretesting of new questions, experiments on question wording and satisficing, incentive approaches, contact modes, and other studies. Substantive questions covered various topics related to family, religion and values amongst others from established cross-national studies such as the Generations and Gender Programme (GGP), the European Values Study (EVS) and the International Social Survey Programme (ISSP). The CRONOS data can be analysed together with responses from the ESS face-to-face interview, allowing for new substantive and methodological analyses which was not previously possible.
This session invites papers on methodological findings from the panel. Papers might cover: representativeness, cost analyses, contact mode effects and incentive strategies, effects of device on measurement, efforts to improve survey completion respondent behaviour and the impact of including off-liners through device provision.
Keywords: CRONOS, ESS, online survey panel, methodological experiments, mode effects
Professor Rory Fitzgerald (European Social Survey (ESS) ERIC HQ, City, University of London) - Presenting Author
Dr Gianmaria Bottoni (European Social Survey (ESS) ERIC HQ, City, University of London)
A number of countries in Europe and beyond have established online or mixed mode panels with a web component based upon probability samples. The European Social Survey (ESS) has followed this approach and applied it for the first time in a cross-national environment. Respondents were recruited to an online panel at the end of the main ESS face-to-face interview and then asked to complete bi-monthly interviews on-line over a 12-month period.
The CROss-National Online Survey (CRONOS) represents the world’s first attempt to set up an academically driven, input harmonised, cross-national, probability-based online panel. Our paper aims to assess the representativeness of CRONOS achieved sample compared both to population data and to ESS Round 8. We use the data from the main ESS survey and compare the full ESS sample of respondents to those who took part in the panel. This allows a comparison between the characteristics of the population that participated in the web panel with those who did not whilst avoiding contamination by mode effects.
The paper compares CRONOS, ESS and Population distributions and then tests whether specific demographic variables predict the propensity to join the panel. Finally the paper compares differences between those who took part in the panel and those who did not on key attitudinal variables such as immigration, climate change and attitudes towards gays and lesbians. This represents a unique opportunity to compare characteristics of the web panel sample with the sample from a high quality survey on those variables that are rarely available in official surveys or where comparability is hampered by differences in data format. We conclude the paper providing some practical implications useful to maximise representativeness when setting up a cross-national, probability-based online panel.
Dr Olga Maslovskaya (University of Southampton) - Presenting Author
Dr Peter Lugtig (Utrecht University)
Professor Gabriele Durrant (University of Southampton)
We live in a digital age with high level of use of technologies. Surveys have also started adopting technologies for data collection. There is a move towards online data collection across the world due to falling response rates and due to pressure to reduce costs of surveys. However, evidence is needed to demonstrate that the online data collection strategy will work and will produce reliable data which researchers can confidently use for policy relevant research. No research has been conducted so far to assess non-response bias and representativeness in online probability-based panels. This paper is very timely and will fill this gap in knowledge.
This paper aims to explore representativeness and non-response bias across waves in the first cross-national online probability-based panel (CRONOS). CRONOS panel data were collected on the back of the European Social Survey (ESS) Round 8. After completing the ESS face-to-face interview, respondents in three countries (Estonia, Great Britain and Slovenia) who were 18 years old or older were invited to participate in seven 20-minute online surveys over a time period of twelve months.
We employ R-indicators as well as other indicators to assess representativeness across waves in cross-national context in CRONOS data. The analysis allows comparison of the results over time as well as across three countries. Preliminary results suggest that there are differences in between- and within-country representativeness over time across three countries used for the analysis.
Dr Elena Sommer (Survey of Health, Ageing and Retirement in Europe, Munich Center for the Economics of Aging) - Presenting Author
Dr Gianmaria Bottoni (European Social Survey, City University of London)
With the recent increased spread of smartphones, the proportion of online surveys completion on them has been rapidly growing (Dillman 2017). In the probability-based CROss-National Online Survey (CRONOS) panel recruited at the back of the European Social Survey Round 8 in Great Britain, Estonia and Slovenia, about one third of respondents completed online surveys using a smartphone. Several studies reported a lower quality of survey data collected via mobile devices as compared to desktop computers (e.g. Mavletova 2013; Struminskaya et al. 2015). However, more recent research has demonstrated that smartphone survey completion does not necessarily result in lower data quality (e.g. Antoun et al. 2017; Mattews et al. 2018). Reflecting on the increased smartphone use in the general population in terms of time and familiarity, the CRONOS panel has been designed as a smartphone-friendly online panel taking into consideration potential challenges of smartphone completion that can have an impact on substantive findings. In our presentation, we first present the smartphone-friendly questionnaire design approach implemented in CRONOS (e.g. visual design of questions and response categories). Then, we compare respondents who completed CRONOS surveys on a smartphone with respondents who completed them on a desktop computer in terms of their demographic characteristics, self-reported multitasking, locality and presence of other people during survey completion as well as respondents’ evaluation of the survey. In the last section of our contribution, we compare smartphone respondents with desktop respondents with regard to several data quality indicators. Finally, we discuss survey items that were associated with high break-off and non-response among smartphone respondents identifying different patterns that result in quality differences across device modes. This could be beneficial for guiding survey practitioners aiming at designing high quality online surveys.
Dr Elissa Sibley (City, University of London) - Presenting Author
Professor Rory Fitzgerald (City University, London)
There is a growing interest in using representative probability-based online panels in place of, or to complement, face-to-face surveys. It is therefore important to establish whether the substantive findings generated by such web panels mirror those obtained via high-quality face-to-face surveys. If the findings differ, understanding why they don’t produce the same estimates will be important.
The CRONOS web panel was fielded in Estonia, Slovenia and Great Britain as a follow-up to Round 8 of the European Social Survey (ESS). It is the world’s first probability-based online panel that uses input harmonised methods. Those who did not have access to the internet were offered an internet-enabled tablet for the duration of the panel.
This paper uses data from the main face-to-face ESS questionnaire to compare the predictive models obtained for three attitudinal outcomes (social trust, attitudes towards homosexuality and attitudes towards immigration) when using the entire ESS sample versus only the subsample of respondents who later participated in the CRONOS web panel. We then run these analyses again excluding those panel participants who had been provided with a tablet, to see whether the absence of the offliners has an impact on the models.