Fielding Cross-Cultural General Population Surveys on the Web? Interactions of Methodological and Substantial Issues in EVS 2017 |
|
| Session Organisers | Dr Michael Ochsner (FORS) Professor Ferruccio Biolcati (University of Milan) |
| Time | Friday 9 July, 16:45 - 18:00 |
Cross-cultural general population surveys, such as the EVS, WVS, ESS, ISSP are an important cornerstone for comparative research and rely on encompassing face-to-face (f2f) interviews. Such a data collection mode has been challenged by dropping response rates and increasing survey costs, and more recently by the sanitary restrictions due to the Covid-19 pandemic. In this session, we present results from a comprehensive experiment using the EVS 2017 to investigate the viability to switch cross-cultural surveys to self-administered web/paper mixed-mode. Such a change raises several methodological questions, such as data quality and comparability across time. The presentations address methodological issues of the comparability across different designs, such as potential selection and order effects, and how to tackle the problem of missing data in matrix designs from a substantive perspective using real-life data and research questions.
Keywords: Matrix design, mixed-mode; European Values Study (EVS); cross-cultural surveys; general population surveys
Professor Ferruccio Biolcati (University of Milan) - Presenting Author
Dr Riccardo Ladini (University of Milan)
Professor Francesco Molteni (University of Milan)
Professor Andrea Pedrazzani (University of Milan)
Professor Cristiano Vezzoni (University of Milan)
When analyzing the relationship between individual religiosity and attitudes toward immigration, empirical research often leads to contrasting findings. Our contribution aims at providing evidence toward the two effects that religiosity can exert on attitudes toward immigration. On the one hand, belonging to a religious community could reinforce a social identity, leading to negative attitudes toward outgroups. On the other hand, religious commitment implies adhering to religious teachings which promote altruistic values, such as the acceptance of others. Since the two hypotheses pertain to different dimensions of religiosity, we argue that the use of a typology which combines religious affiliation and church attendance allows testing the two hypotheses as complementary. In a previous work focusing on contemporary Italy, we provide first evidence to the hypotheses, by showing that non-religious and highly-religious people show more positive attitudes toward immigration than affiliated individuals with low religious commitment.
The presentation aims at further investigating the relationship by exploiting the EVS 2017 mixed-mode innovations. Survey research is currently afflicted by declining response rates and increasing data collection costs. Mixed-mode and related innovations appear to be among the most effective strategies in dealing with these challenges. In the context of EVS 2017 data collection, six countries (Denmark, Finland, Germany, Iceland, the Netherlands, Switzerland) applied innovative mixed-mode research designs.
Among the countries, there is some variation in the research design. Beyond face-to-face interviews, online and postal paper-and-pencil modes were applied. Within the self-administered mode, some countries provided the full-length questionnaire in the original or modified order. Some countries chunked the questionnaire according to a matrix design with or without a follow-up. Finally, different contact modes, sampling and incentives were used.
These innovations provide plenty of opportunities for methodological analysis. The results of different response rates and selection processes may be assessed. At the same time, we may take into account mode and order effects as well as context effects (questions about religious affiliation and church attendance were administered together with other religious questions in some designs and alone in other designs). As well, to measure attitudes toward immigration EVS includes one scale and multiple items. Moreover, matrix design involves a substantive share of missingness by design to deal with.
For all these analyses, EVS longitudinal data may serve as a useful benchmark. Moreover, the selection of countries involved in the innovative designs gives us the opportunity to extend our research question from Catholic countries like Italy and Ireland to countries with a protestant or a mixed religious tradition.
Finally, the focus on a substantive research question (the multifaceted relationship between individual religiosity and attitudes toward immigration) offers the opportunity to work at two different levels. Beyond the level of the single variables, we may assess the possible effects of mixed-mode design for the overall relationship. A priori it should not be ruled out that differences at the variables level vanish when we step up to the relationship level.
Dr Vera Lomazzi (GESIS Leibniz Institute for Social Sciences) - Presenting Author
The implementation of mixed-mode surveys and matrix designs are relevant innovations that can support the development of survey research in the future. Being cost-effective, less time demanding for the respondents, and potentially decreasing coverage errors and non-response errors, these solutions appear promising. However, what are the practical consequences of using these techniques for data comparability and what should survey designers and data users take into account?
The use of mixed-mode can introduce a method bias threatening the measurement equivalence across modes and cultural context. For example, different survey situations, ways of providing items to the respondent, the layout on the screen, or slight changes in the wording of items across modes of data collection can affect the response process and lead to non-equivalent measurement constructs between modes.
When dealing with cross-national surveys, these biases could be amplified by cultural differences in item interpretation, translation issues, etc. and cross-cultural comparisons may be affected. Using non-equivalent measures can bring scholars to misleading substantive results because differences in measurement characteristics may be taken for real differences in the underlying constructs.
The current study considers mixed-mode as potential sources of method bias, which can lead to measurement non-equivalence and it uses the measurement of gender role attitudes included in the fifth wave of the European Values Study as a working example.
Gender role attitudes refer to what is believed appropriate for male and female roles. Several survey programmes include measurements of gender role attitude but, despite their large use in substantive research, such measurement is often problematic also because they are often particularly sensitive to cultural biases. Criticisms range from the inadequacy of the instruments, which often neglect the multidimensional nature of this concept; the strong accent only on female roles; the main focus on the private sphere overlooking attitudes towards roles in the public realm; outdated formulations; the lack of measurement equivalence in cross-cultural settings.
Previous research based on the scale implemented in the previous waves of EVS, WVS and ISSP addressed the change of a new scale used in the recent EVS2017. Even if the item formulations could still be considered a bit outdated and the focus is kept on women’s roles, the new scale has improved content validity. However, the implementation of mixed-mode can further challenge its comparability.
By using Multi-Group Confirmatory Factor Analysis and the alignment method, the study shows that the EVS gender role attitudes scale is comparable across countries and modes of data collection.
Dr Michael Ochsner (FORS) - Presenting Author
Several developments challenge the face-to-face gold-standard for cross-cultural general population surveys: a more active lifestyle and urbanisation lead to decreasing contact rates and the Covid-19 pandemic makes face-to-face impossible or at least adds insecurity, both leading to higher costs for already expensive face-to-face surveys. Web/paper surveys thus attract the attention of survey practitioners even more given increasing internet penetration rates and the habit of doing things online during confinement. However, cross-cultural general population surveys cover a broad range of topics and thus interviews usually last one hour or even more. Yet, research on web surveys suggest a maximum duration for web surveys of 20-30 minutes. Reducing the number of questions covered in the survey is not an option as the advantage of cross-cultural general population surveys is the breadth of topics making it possible to study a large number of diverse research questions. One solution is to split the questionnaire into multiple sets and ask each respondent only a part of the questions, i.e. implementing a matrix design. Given the random attribution of the groups, the missing values due to questions not asked in one group but in another are completely at random and, therefore, researchers can still analyse many topics provided that the questions used are fielded in the same groups.
However, what if a researcher is interested in using questions that are spread over several or even all splits? In such a case, the researcher ends up with a model with very few or zero observations as only a few or even no respondent has answered all questions.
In this presentation, I will suggest multiple imputation as a solution to this problem. I will use the EVS 2017 mixed-mode experiment data where different designs of web/paper mixed-mode surveys were conducted along with the standard face-to-face mode. I will compare differences across designs with regard to results of a substantive example that includes many variables and thus is a strong test to comparability across designs and a particular challenge for matrix designs. The example examines the research question whether the state has a social responsibility. The last decades have been dominated by austerity politics and welfare services have been cut back or transformed. Some theories suggest that this might come with a risk for undermining the cohesion in society (Heimann, 1921) while others claim that power relations and meritocratic attitudes lead to acceptance of inequalities (Bourdieu, 1985) or that it depends on values put forward in a society (Esping-Andersen, 1990). The question therefore arises whether a retrenchment comes with a risk of diminishing legitimacy of the state. I examine whether persons who evaluate the social security system of the state positively perceive the state as more legitimate than those who evaluate the social security system less favourably.
Professor Tim Reeskens (Tilburg University) - Presenting Author
Professor Yvette van Osch (Tilburg University)
In an impressive large-scale cross-cultural study, Van Bavel and colleagues (2020) explore what predicts health behavior during the COVID-19 pandemic. Their focus on national identity reveals that the more strongly people identify with their nation, the more likely they are to adhere to health regulations and support public health policies. Their findings suggest that governments and leaders can speak to, or are even able to create a strong sense of in-group sentiment of love towards compatriots, thereby influencing the adherence to important policies to constrain the spread of the virus and consequently successfully combatting the coronavirus crisis.
One non-negligible limitation of their research is the cross-sectional nature of the data. The aim of our contribution is to replicate their findings by analyzing data from the European Values Study, focusing on the effect of national identity on trust in the health care system. First, we analyze the cross-national cross-sectional data from the European Values Study (2017) to analyze the robustness of this effect from national identity on trust in the health care system. Second, we causally leverage this nexus by zooming in on the Netherlands, where the mixed mode fieldwork of 2017 allowed for the creation of an web panel representative of the Netherlands. Participants of this panel have been re-approached amidst the first wave (May 2020) and the second wave (October 2020) of the pandemic. By using panel regression, we engage with the question whether increases in national identity translate into more trust in the health care system, as Van Bavel and colleagues (2020) propose.