Measurement Invariance: Testing for it and Explaining Why It Is Absent 1 |
|
| Chair | Dr Katharina Meitinger (GESIS Leibniz Institute for the Social Sciences ) |
| Coordinator 1 | Professor Eldad Davidov (University of Cologne and University of Zurich) |
In previous research at occasion of the Belgian General Election Studies between 1991 and 2007 it was found, and replicated, that a bipolar national identity latent variable was an equivalent and valid measurement of the intensity of (sub)national feelings in both the Flemish and Walloon regions. It was shown that the attitude towards foreigners (ethnic threat) and the (sub)national consciousness variables were inversely related in Flanders and Wallonia. The over-time measured latent variable of (sub)national consciousness contained a mix of indicators on state reform and on (sub)national identity with comparable high factor loadings. These findings support the hypothesis that the relationship between (sub)national identity and attitudes towards foreigners is not intrinsic but depends on social and historical factors.
One particular cultural explanation is that the inversed relation between the two latent variables depends partially on the social representation of the nation. This hypothesis was however not tested because of lack of measures of the latter. Later studies based on the 2003 ISSP module on national identity supported this proposed explanation, and demonstrated empirically that the variability in the relationship between ethnic prejudice and national identity is partially accounted for by the various ways in which nations are defined (Pehrson & Vignoles, 2009: 24, 33). This suggests that measures of “collective definitions of the nation” (representations) may moderate the relationship between ethnic prejudice and (sub)national identity.
In order to test this hypothesis, in the 2014 ISPO election study, direct measurements of civic and ethnic citizenship conceptions were included as individual level proxy variables for the theoretical concept ‘representation of the nation’. However, it was no longer possible to measure in the 2014 Walloon sample our four item (sub)national consciousness variable on an identical way as it was in all previous ISPO samples, and as it is still the case in the Flemish sample. For the first time the two (sub)national identity indicators seem to measure a different concept than the two state reform indicators. In this paper, we investigate the issue how this change in measurement equivalence of the latent variable in Wallonia between 2007 and 2014 ISPO election surveys can be explained? It is suggested that substantial change in the support for a sub-nationalist (or regionalist) party in one region (Flanders) might have an effect on the measurement of (sub)national consciousness in the other region (Wallonia). Methodologically, this new insight result in the idea that conclusions about invariant measurement may depend on the social context, meaning that even in 2007 the strictly invariant measurement of (sub)national identity might have got already a different social meaning. We will discuss the potential ways of dealing with the general idea that measured concepts might have different meaning even when the indicators met the criteria of scalar invariance.
References
Maddens, B., Billiet, J., Beerten, R. (2000). National identity and the attitude towards foreigners in multi-national states: - the case of Belgium. Journal of Ethnic and Migration Studies, 26 (1), 45-60.
Billiet, J.,
Detecting and explaining potential sources of bias in cross-cultural research is cursory to reach measurement invariance. The lack of a comprehensive and systematic approach is particularly serious when different linguistic versions of psychological or health scales are included in survey questionnaires. The aim of the paper is twofold: a) to present an integrated approach to detect and explain bias from a mixed-methods research framework; and b) to illustrate how qualitative evidence from cognitive interviewing along with quantitative results can improve our understanding of measurement inequivalence. We review current consensus about how to reach integration in a measurement equivalence research through the three main integration levels: design, method, and interpretation and reporting. Paying much more attention to qualitative evidence of different sources of bias, we support our approach to measurement equivalence with a cross-cultural empirical research on quality-of-life. CI was conducted with participants from the Netherlands and Spain who responded to Quality of Life (QoL) items and questions included in major international survey research projects. Qualitative findings from CI provide insight into sources of measurement inequivalence related to item wordings, interpretation patterns, and differential understanding and use of the response options. Results of the study support the contribution and usefulness of CI to informing about the three types of bias within a mixed methods research framework.
Introduction: Measurement non-invariance is a frequently encountered problem in comparative research on migrants. While sophisticated techniques are available to account for measurement non-invariance in the analysis phase of a study, little is known about its underlying causes. In this paper, results from two comparative surveys are presented. The first investigation compares illness perceptions between Turks in Turkey and Turkish migrants in Germany, the second examines health-related quality of life (HRQOL) in the general population in Germany comparing migrants and non-migrants. A mixed-method approach along a framework of functional equivalence is used to examine causes of measurement non-invariance.
Methods: In the first study, 602 patients of Turkish descent in Germany and Turkey were surveyed on their illness perceptions by means of the 38-item Revised Illness Perceptions Questionnaire (IPQR) administered through standardized Turkish language interviews. For the second study, data on 22,050 subjects are available who, amongst others, were surveyed in German language on their HRQOL using two subscales of the SF36 (vitality and mental health, consisting of 4 and 5 items, respectively). In both studies, measurement non-invariance across groups (Turkish migrants in Germany vs. Turks in Turkey and migrants vs. non-migrants in Germany, respectively) is examined through multiple indicators multiple causes models (MIMIC) and multi-group confirmatory factor analysis (MGCFA). Causes for measurement non-invariance are examined along conceptual, operational and sematic dimensions by means of a systematic literature review, cognitive interviews with respondents and interviews with experts.
Results: The IPQR showed configural non-invariance between Turks in Turkey and Turkish migrants in Germany with respect to 8 items. Both the IPQR and SF36 items showed differential item functioning (DIF) across groups in several items. Cognitive studies reveal that a number of the non-invariant items were misunderstood by migrants because of complex item wording or unfamiliar phrases used throughout the questionnaires. Expert interviews and a systematic review suggest that language attrition and change leading to differences in language usage may be responsible for measurement non-invariance when migrants are surveyed by means of questionnaires developed or adapted for the population of the respective countries which they originate from. MIMIC models further reveal that DIF related to group status is confounded or moderated by sex, age and socioeconomic status in some items.
Conclusion: This paper illustrates that measurement non-invariance encountered in surveys involving migrants may have different causes. Because of language attrition and change, ‘mother tongue’ questionnaires developed or adapted for the population of the respective countries of origin may perform differently in migrants. DIF related to group status may also be confounded by or may vary across the values of a third variable. In other instances, also conceptual differences and differences in operational aspects of surveys need to be considered. A mixed-method approach along an analytical framework of functional equivalence can help to identify causes of measurement non-invariance and can guide appropriate measures aiming to address non-invariance in different phases of the research process.
Measurement invariance is key in ensuring that a latent construct across countries has the same meaning. However, it is often the case that latent constructs diverge cross-nationally (the more countries in the sample the more likely there is measurement variance) and scholars lack established quantitative tools to assess the reasons behind these divergences (cf. Davidov et al 2014). On the example of welfare state solidarity in 13 European countries the paper looks at how the concept and meaning of welfare state solidarity varies country to country and explores the reasons behind these trends. The paper offers a two-step quantitative approach to interpret the measurement variance present in social surveys across countries, where in the first step multi-group confirmatory factor analysis is applied, and in the second established network analysis techniques. After establishing configural invariance with exploratory and confirmatory factor analysis across countries, but no structural invariance across can be determined for all 13 countries with MGCFA. However, instead aiming to establish at least partial invariance across countries by setting some factor loadings free, the paper conducts MGFCA on country pairs. The fit indexes (and to some degree factor loadings) of the MGCFAs are then interpreted as to express the distance between the meaning of the concept welfare state solidarity between two countries, thus creating dyads. The dyads then can be formed to create a closed network, where the countries are the nodes and the ties between the nodes are the fit indices. So in a second step the effects of various cultural, economic, or historical factors are tested by means of network analysis. Hence, this way one can get a better understanding of eventual substantial reasons for explaining non-invariance.