All time references are in CEST
The International Social Survey Programme (ISSP) - data quality issues in cross-national perspective 1 |
|
| Session Organisers |
Professor Stephanie Steinmetz (FORS and University of Lausanne) Dr René Bautista (NORC at the University of Chicago) Dr Evi Scholz (GESIS Leibniz Institute for the Social Sciences) Professor Markus Hadler (University of Graz) |
| Time | Thursday 20 July, 14:00 - 15:30 |
| Room | U6-02 |
In an increasingly interdependent world, to learn about the attitudes and behaviour of the population around the globe is essential. While many cross-national surveys are exclusively one-shot or one-topic initiatives, continuous global social surveys with high-quality standards are rare. The International Social Survey Programme (ISSP) is one of the exceptionally large and continuous programmes designed to offer both cross-national data and long time series. Since 1985, the ISSP conducts annual surveys on various topics important for social science research. Methodological issues are discussed in an extra committee of ISSP experts where national sample designs are checked for compliance with ISSP requirements and signed off for individual ISSP members. The ISSP is also exceptional as an organisation based on democratic cooperation of equal partners, which offers a special atmosphere for all kind of discussion about various subjects and flexibility in terms of adapting to new challenges and unforeseen issues. In addition, ISSP data, questionnaires and documentation, e.g., on data collection or on questionnaire development, are free of charge for all researchers interested in cross-national data. The ISSP is thus a priceless resource for various research areas in need of global survey data with high methodological standards and high quality data.
This session aims to provide and foster the exchange about the ISSP among providers and interested researchers already using or planning to use the ISSP data in their research. In the session, we will focus in particular on methodological challenges related to the data quality of cross-national surveys, however also substantial papers using one or various ISSP modules are welcome to the session.
Keywords: ISSP, cross-cultural research, comparability, data quality, measurement, survey mode, questionnaire development, cognitive testing
Dr Rene Bautista (NORC at the University of Chicago) - Presenting Author
Mr Benjamin Schapiro (NORC at the University of Chicago)
Cognitive testing is a helpful pre-testing technique for survey questions. It is aimed to identify issues related to understanding and interpretation of words and phrasing used in a survey question, matching social constructs more carefully to new survey items. Also, it helps to identify issues related to memory retrieval (i.e., memory erosion), rationale and decision making when determining an answer, and assessing how respondents map their thinking into response categories. The International Social Survey Programme (ISSP) has developed a new survey module (National Identity and Citizenship) to be implemented in 2023 which covers a variety of relevant topics, including questions on populism. These questions will be fielded across the globe by ISSP members. During the development of this new module, the USA team conducted cognitive interviews on newly designed questions for the module, focusing mainly on the understanding of populism. American respondents shared their understanding of the will of the people, ordinary people vs. politicians, the inaction of politicians, political compromise, perceptions of politicians, elitism, and perceptions of economic policy. The issues raised in this cognitive testing were integral to the development of the final module. This presentation will discuss results from the cognitive interviews as well as other testing undertaken by the drafting group, with an emphasis on how cognitive interviewing provided a new dimension of evaluation for new-to-ISSP items. We will also discuss the implications for survey measurement properties identified through cognitive testing, and offer thoughts on how to expand testing in ISSP drafting groups.
Mr Arturo Bertero (University of Milan)
Mr Gonzalo Franetovic (University of Milan) - Presenting Author
Inequality beliefs refer to the different explanations that people give to the unequal distribution of resources within a society. Scholars are used to distinguish between individualist beliefs -when inequality is perceived as a result of individual attributes, like education and hard work- and structuralist beliefs -when it is associated to determinants beyond people’s control, like race or sex. However, this categorization is derived from factor analytic techniques, which were repeatedly found to underestimate the number of factors underlying data, especially when highly correlated. Hence, this paper seeks to answer the following research question: how are inequality beliefs structured across contemporary societies?
We intend to answer it by adopting a complex system perspective originated from network psychometrics, which conceptualize an attitude as a network in which nodes represent attitude elements that are connected by direct causal interactions. The following hypotheses will be tested:
H1: The inequality beliefs battery will show more than two dimensions in the majority of the ISSP countries.
H2: The items composing the individualist beliefs dimension will be more central in the attitude networks of countries characterized by high GINI.
To test our hypotheses, we will use the inequality beliefs battery (variables Q1a to Q1j) from the 2019 ISSP Module of Social Inequality and all 29 countries of the sample will be analyzed. The alleged two-dimensionality of the inequality beliefs battery will be investigated trough an innovative technique called Exploratory Graph Analysis (EGA).
Our preliminary results suggest that inequality beliefs are clustered in more than two components in all but three countries. EGA seems to indicate that besides the classical distinction between individualists and structuralists beliefs, a political dimension is present. Our contribution represents the first application of EGA to the inequality beliefs battery in a comparative perspective.
Professor Renzo Carriero (University of Turin) - Presenting Author
Questions on actual and ideal wages of various occupations are distinctive features of ISSP Social Inequality modules. They represent meaningful ways to quantify respondents’ perceived and tolerated wage inequality, by (ex-post) computing pay ratios. However, recent research (Pedersen & Mutz 2019; Heiserman & Simpson 2021) raised heavy criticism to pay ratios, highlighting that they suffer from anchoring bias, ratio bias, and low validity more generally. In this contribution I investigate whether it is possible to handle actual and ideal pay ratios in such a way that they provide meaningful and accurate measures of individuals’ attitudes toward wage inequality, especially in the case of comparative cross-country analysis. I tested several ways to adjust ideal pay ratios to actual (i.e. perceived) pay ratios, in single-country and pooled data analyses. I found that country comparisons may be tricky and very sensitive to adjustments, with differences in ideal pay ratios across countries not matching differences in word-based attitudes. Moreover, two standard predictors of attitudes toward income inequality like party voted and social class are more strongly associated with word-based attitudes rather than numerical indicators such as pay ratios. These results suggest that pay ratios are problematic measures of attitudes towards income inequalities in a comparative framework and perhaps should be replace by more standard word-based items. However, attitudes expressed by word-based items do not measure the extent of inequalities that respondents are willing to tolerate. Therefore, further research is needed to find out ways to fix problems with pay ratios.
Mrs Marta Moroni (Department of Social and Political Sciences, University of Milano) - Presenting Author
Dr Giulia Dotti Sani (Department of Social and Political Sciences, University of Milano)
Dr Riccardo Ladini (Department of Social and Political Sciences, University of Milano)
The environmental problem is one of the main challenges that our society is living and it is currently growing in visibility and salience. Cumulative evidence shows an increase in climate change awareness and environmental concern over time and across populations. In the common narrative, the rise in environmental concern is often associated with a higher awareness of younger generations that are generally viewed as more concerned than older ones about protecting the environment. This sentiment was made more salient after the global success of Greta Thunberg’s movement Fridays for Future, which mostly involves very young people.
But is there empirical evidence of a generational gap in environmental concern? And to what extent does this concern translate into willingness to pay for the environment? As per the former, existing research does not provide strong evidence regarding the effect of age; for the latter, an overall increase in environmental concern does not seem to be associated with a higher willingness to pay (i.e. to contribute economically, for example through higher taxes) to protect the environment.
Thus, this research explores the relationship between environmental concern and the willingness to pay using especially the 2020 “Environment IV” of the International Social Survey Programme. We investigate, first, the extent to which young and older people differ in their attitudes toward the environment and willingness to pay to protect it; second, we analyse the relationship between these two dimensions across countries, also by examining the role of contextual characteristics; lastly, we investigate whether this relationship differs between people of different age groups within countries.