All time references are in CEST
Adapting survey mode in a changing survey landscape: Experiences from repeat cross-national, cross-sectional, and general social surveys 2 |
|
| Session Organisers | Ms Siobhan O'Muircheartaigh (European Social Survey ERIC (City, University of London)) Mr Tim Hanson (European Social Survey ERIC (City, University of London)) Dr Oshrat Hochman (GESIS) Dr Rene Bautista (NORC at the University of Chicago) Professor Rory Fitzgerald (European Social Survey ERIC (City, University of London)) |
| Time | Tuesday 18 July, 14:00 - 15:30 |
| Room | U6-11 |
Studies to measure attitudes, opinions and behaviours have been and remain critical to understanding societies around the world. In the face of the COVID19 pandemic and changing trends in the interviewer workforce, many repeat cross-sectional social surveys have been experimenting with self-completion and mixed-mode approaches. The European Social Survey (launched in 2001) and United States’ General Social Survey (launched in 1972) are key examples of long-standing studies that collect data to inform research on changes over time, and are now exploring new modes for the future. This session brings together ESS, GSS, and other cross-sectional social surveys to share experiences in survey mode transition.
The session's aims are threefold: (1) Share strategies and lessons-learned from recent mode experiments by ESS, GSS, and other studies, and potential ways to improve methods in future. (2) Highlight how different cross-sectional studies have modified survey protocols in recent years to adapt to changing conditions in the public (e.g., public health crisis, shifting communication modes, public’s willingness to respond to surveys, trends in interviewer workforce). (3) Provide a space for data creators, data users, and survey practitioners to discuss methodological and statistical challenges for cross-sectional studies considering such a move.
We invite submissions from those involved in transitioning repeat, cross-sectional, and cross-national social surveys to new data collection approaches. Topics of interest include: results from pilots or feasibility studies based on self-completion or mixed-mode approaches; findings from experimental research testing aspects of self-completion/mixed-mode designs (e.g., incentive and mailing strategies, survey length adaptations, sequential vs. concurrent designs); impacts of mode switches on measurement and survey time series; and discussions of experiences and challenges associated with adapting cross-sectional surveys to new modes across different cultural/national contexts.
Ms Mónica Méndez Lago (CENTRO DE INVESTIGACIONES SOCIOLÓGICAS) - Presenting Author
The presentation will focus on the shift from a face-to-face to a self-completion mode design in the implementation in Spain of the 10th round of the European Social Survey (2022) and the Environment ISSP module (carried out in 2022-23). These two surveys will be compared with the Push-to-Web/+Mail survey carried out in Spain, paralell to the ISSP 2014 face-to-face survey. The protocol used for the three surveys is very similar, but there are small variations, such as the shorter length of the ISSP questionnaire or some differences in the mailing/incentive strategies used, that render the comparison more interesting.
The presentation will analyse: 1) the mailing protocol/strategy used in the three surveys, which is slightly different both in terms of timing and when the paper questionnaire was included; 2) the incentive strategy used, also with small variations across the three surveys 3) the response rate in the Web/Mail phase, looking at the socio-demographic profile of respondents, comparing it with the results in the most recent F2F implementations of the different surveys analysed). It will also look at the extent to which non-Internet users (or less intensive users) are reached with each method. The comparison of the more recent ESS and ISSP surveys with the 2014 ISSP module is particularly interesting, since this will make it possible to analyse how response rate and the representativeness of final achieved samples has evolved over time.
In sum, the aim is to present an overview of the shift from face-to-face to a Push-to-Web&Mail mode in the implementation of two important repeat cross-national programs, the ESS and ISSP, and to reflect on the lessons learned from these three experiences for future surveys.
Mrs Caroline Hahn (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Dr Jan-Lucas Schanze (GESIS - Leibniz Institute for the Social Sciences)
Mr Niklas Donth (GESIS - Leibniz Institute for the Social Sciences)
Mrs Oshrat Hochman (GESIS - Leibniz Institute for the Social Sciences)
The European Social Survey (ESS) is based on rigorous state of the art methodological standards, in particular in terms of comparativeness. The ESS design requirements were challenged when the corona pandemic hit the world. The necessary reduction of social contacts caused difficulties for the traditional face-to-face mode used exclusively in the ESS until then. Nine countries participating in the ESS decided to switch their survey mode from face-to-face to self-completion modes (CAWI and PAPI) for ESS Round 10, among them Germany. This switch of modes implied a stark deviation from previous rounds of the ESS and a highly irregular variability in ESS data collection.
The consequences of the mode switch for cross-time as well as cross-country comparability require a thorough investigation. This paper focuses on a comparison of a wide range of quality measures obtained in the German self-completion ESS with the previous ESS round implemented in Germany in the face-to-face mode (ESS Round 9). We investigate the impact of a mode switch for response rate, sample composition (e.g., age, level of education, migration background), and data quality (e.g., item nonresponse, filter errors, response patterns). The paper differentiates results by web mode and paper mode to test whether the disadvantages associated with the use of paper questionnaires are compensated by their contribution to the general response rate and to the sample composition. We also present results from a contact experiment designed to understand whether older cohorts (60 or older) are equally willing to participate in the survey using an online questionnaire rather than a paper questionnaire. For this purpose, we provided half of this group with a paper questionnaire only in the second reminder, while the other half received the paper questionnaire in the first advance letter.
Professor Heikki Ervasti (University of Turku)
Dr Takis Venetoklis (University of Turku)
Mr Sami Mustikkamaa (University of Turku) - Presenting Author
Ms Mari Anttila (University of Turku)
Despite ever-growing reliance on survey data and an increasing shift to self-completion formats, little research exists on mode effects between face-to-face and self-completion surveys. We present results from a nationally representative, population-based mode experiment. The experiment was conducted during and after the Round 10 fieldwork of the European Social Survey in Finland. Right after the regular face-to-face fieldwork of the ESS Round 10 in late 2021, a self-completion version of the same questionnaire was fielded in early 2022 using a similar random sample of the Finnish population. We analyze the data from the perspective of social desirability bias, satisficing, and other sources of potential mode effects. Our preliminary results indicate similar response rates and very little differences in representativeness of the data between the two modes. However, large mode effects appear in certain often-used items, such as questions on social and political trust. Other items, such as the political left-right position of a respondent, show little mode differences. The observed mode effects pose considerable challenges for survey research in the future.
Ms Siobhan O'Muircheartaigh (European Social Survey ERIC (City, University of London)) - Presenting Author
Mr Tim Hanson (European Social Survey ERIC (City, University of London))
Professor Rory Fitzgerald (European Social Survey ERIC (City, University of London))
This paper will discuss the design, implementation, and results of a self-completion data collection approach by the cross-national European Social Survey (ESS) in 2021/2022. The lessons learned from implementing this approach in 11 countries suggest strategies for improving self-completion methods for the ESS and other general social surveys.
The ESS is an academically-driven survey that has measured attitudes and behavior across Europe every two years since 2002. In the survey’s 10th round, a self-completion data collection method was developed as an alternative approach for countries where the traditional ESS face-to-face methodology was impossible due to the COVID-19 pandemic. The standardized, push-to-web approach combined web and paper self-completion modes. Eleven of 31 countries applied this, with some national variation and experimentation (e.g., sequential vs. concurrent web and paper delivery, postal- vs. fieldworker-assisted requirement, monetary vs. nonmonetary incentives, inclusion of nonresponse mop-up phases). This paper collates national and central management experiences to provide holistic lessons-learned.
This paper will 1) describe the piloting and development of the self-completion approach for ESS Round 10, 2) chart challenges and successes in its implementation cross-nationally during the round, 3) share comparisons between face-to-face and self-completion results, and 4) reflect on future development and improvement, for the ESS and for large-scale cross-sectional surveys more broadly.
Professor Sergio Mauceri (Sapienza - University of Rome)
Professor Maria Paola Faggiano (Sapienza - University of Rome)
Professor Barbara Sonzogni (Sapienza - University of Rome)
Dr Maria Dentale (Sapienza - University of Rome)
Dr Michela Cavagnuolo (Sapienza - University of Rome) - Presenting Author
Dr Viviana Capozza (Sapienza - University of Rome)
Feelings such as distrust in the vaccine production process, fear of unexpected side effects, ideological aversion and social springs that trigger and/or feed them are widely examined in recent scientific contributions on vaccination hesitancy. Our proposal intends to discuss the results obtained through a panel web survey (sample: 2,787 cases), carried out in Italy in two research appointments (2020/2021). The survey allowed us to reconstruct the dynamics of choice related to the sphere of vaccination, with particular regard to anti-vaxxers (234 cases). Three profiles are specifically characterizable: i. the negationist-trivializer, who does not believe in the severity of the disease/in the existence of Covid-19 and who questions the efficacy of vaccines; ii. the distrustful-fearful, who is worried about the possible repercussions of the vaccine on his/her health; iii. the conformist, who approaches vaccination by a forced choice. Through the adoption of a longitudinal design and the application of multivariate analysis, this sample was positioned on the axes of responsibility/distrust and acceptance/rejection of the anti-contagion rules. Following the panel web survey, the inquiry provided for the realization of 96 focused interviews, to be addressed, in an equally distributed way, to the three profiles that emerged, in order to deepen the generative mechanisms related to the reluctance to vaccinate and to further articulate the typology of Anti-vaxxer. Within this Mixed Methods perspective, the theoretical-interpretative framework of hesitant behavior was articulated and strengthened, reaching a deeper understanding of the constellation of practices, social representations, motivations, attitudes connected with the decision not to vaccinate, or to vaccinate due to dynamics of social pressure/coercion.