All time references are in CEST
Measuring Discrimination: Methodological Challenges and Insights |
|
| Session Organisers | Mr Kien Tran (German Centre for Integration and Migration Research) Mrs Almuth Lietz (German Centre for Integration and Migration Research) Mrs Zaza Zindel (German Centre for Integration and Migration Research) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Discrimination based on race, ethnicity, and other characteristics remains a pervasive issue in many societies, contributing to deep-rooted social inequalities. The empirical study of discrimination, especially among racialized and marginalized groups, offers crucial insights into these inequalities but also presents significant methodological challenges. This session will focus on the empirical investigation of discrimination, with a special emphasis on racialized and marginalized populations. The goal is to bring together papers that explore new advancements in survey research related to discrimination reporting.
Survey research is a key catalyst for investigating discrimination empirically. The measurement of discrimination experiences has evolved over time, reflecting shifts in societal attitudes and methodological advancements. How vulnerable group membership is defined and operationalized can influence the prevalence of reported discrimination experiences. Whether individuals are categorized based on self-identification, external perception, or other criteria can significantly affect estimates of discrimination prevalence. Longitudinal trends in discrimination and the development of new techniques for capturing these experiences are critical for understanding how discrimination evolves over time. Experimental designs that simulate discriminatory contexts offer valuable insights into implicit and explicit biases. Furthermore, exploring the relationship between self-reported reasons for discrimination (e.g., race, gender, religion) and individuals' self-identification with vulnerable groups can reveal important discrepancies between perceived and actual experiences of discrimination. These dynamics provide a deeper understanding of how individuals interpret and frame their experiences within broader social contexts.
Relevant topics for this session include, but are not limited to:
• Methodological advances in measuring discrimination, including implicit measures, data-linkage procedures, or comparisons of survey instruments assessing discriminatory experiences.
• Experimental designs simulating discriminatory contexts.
• The influence of operationalization of key characteristics, such as vulnerable group membership and self-identification, on reported discrimination.
• Longitudinal studies of discrimination: trends over time, shifts in prevalence, changes in forms of discrimination.
Keywords: discrimination, racialized groups, measurement, operationalization, racialized categorizations
Mr Jonas Koehler (DeZIM-Institute) - Presenting Author
The operationalization and measurement of discrimination experiences pose a non-trivial challenge in social sciences. Standardized survey instruments, in particular, face critique from qualitative research for reducing complex and intertwined dimensions into a handful of metric variables. To address the topic empirically, a mixed-method design was conducted, combining standardized survey instruments with manually coded data from a qualitative diary study. The analysis of the differing methodologies addresses the question of how standardized and qualitative approaches can both be valid, yet still lead to fundamentally different distributions of discrimination experiences.
Traditionally, surveys rely on standardized single-item indicators or established scales, such as the Perceived Discrimination Scale used in this study. While this provides a time-efficient method to generate metric values for each participant, it offers only a reductionist and aggregated understanding of the multifaceted phenomenon. This is especially true for subtle forms, such as non-verbal discrimination and microaggressions.
Qualitative diary studies follow a different approach, allowing for time-sensitive, granular, and nuanced data collection of everyday experiences (Swim et al., 2003). To systematically analyze and compare the entries, which reflect the participants often ambiguous and fuzzy perception of social reality, the open-text data was manually coded and quantified for this study.
The analyzed data includes 82 participants, surveyed online in the autumn/winter of 2020. The convenience sample was recruited for a project focusing on the experiences of people of Asian origin during the COVID-19 pandemic in Germany. All participants completed the standardized survey as well as the two-week diary study, enabling intra-person comparisons.
Dr Elli Zey (German Centre for Integration and Migration Research) - Presenting Author
Ms Judith Ehmeir (German Centre for Integration and Migration Research)
Dr Iniobong Essien (Leuphana Universität Lüneburg)
Dr Stefanie Hechler (German Centre for Integration and Migration Research)
Dr Susanne Veit (German Centre for Integration and Migration Research)
Survey data on racism and discrimination are often influenced by social desirability bias. Therefore, indirect measures of stereotypes and bias provide a complementary picture by providing insights into the cognitive processes and behavioral tendencies involved. As indirect measures, cognitive tests measure bias by analyzing reaction times, behavior under time pressure, priming effects, and memory effects instead of explicit responses, which are mostly used in surveys. MIND.set provides easy access to create and administer these tests to measure implicit (racial) bias and implement them in online surveys. The platform currently includes five established cognitive tests: Implicit Association Test (IAT), Affect Misattribution Procedure (AMP), First Person Shooter Task (ST), Avoidance Task (AT), and Source Monitoring Paradigm (SMP). The platform prioritizes accessibility in several ways: The platform is free for researchers, from students to professors, and the browser-based application requires no additional software downloads. The versatile and user-friendly interface requires no additional programming skills, allowing researchers to create and monitor tests and download data using a click-based GUI (general user interface). We support users with step-by-step instruction manuals for test creation, a media pool for managing stimulus material, and pre-written commented analysis scripts in R for easy data interpretation. MIND.set is also designed so that participants can access and take the tests via computers or mobile devices such as smartphones. We provide code snippets for seamless integration into various survey systems (e.g. SoSci Survey, Lime Survey, EFS, Unipark), but all tests can also be accessed directly via a standalone link. By addressing the challenges of remote testing with indirect measures and providing comprehensive resources, MIND.set improves the accessibility, reliability and scalability of cognitive tests in online surveys.
Dr Jozef Zagrapan (Institute for Sociology of the Slovak Academy of Sciences) - Presenting Author
Surveys, commonly used to assess public opinions and attitudes, are often presumed to capture the authentic values of the measured concepts. Nevertheless, a substantial body of research challenges this assumption by highlighting the impact of question characteristics on respondents. The influence of question framing on survey responses is a well-established phenomenon, as evidenced by research demonstrating that the decision to phrase questions positively or negatively shapes the answers. This study tests these assumptions and contributes to the existing body of knowledge by examining how framing the statements in positive or negative terms in the Slovak language may impact survey outcomes.
As part of a larger survey (N = 1325), we conducted a split-ballot experiment involving two statements on religion framed both positively and negatively. Respondents provided feedback on a five-point scale, ranging from 'definitely agree' to 'definitely disagree.' In the first scenario, the sole distinction in the statement lay in the inclusion of the word 'not.' Respondents were asked to agree or disagree with the statement 'All religious groups existing in Slovakia should have equal rights' in the positive variant and 'Not all religious groups existing in Slovakia should have equal rights' in the negative variant. In the second case, respondents expressed their agreement or disagreement with the statement 'We need to respect all religions' in the positive version and 'Some religions we do not need to respect' in the negative version. The results reveal differences between framing in both instances. In the negative versions, positive answers (definitely agree + rather agree) prevail in both cases. Moreover, positive answers in negative variants are chosen more frequently than negative answers in the positive variants. Additionally, the middle option ('do not agree nor disagree') is selected less often in negatively framed statements.
Dr Andrea Forster (Utrecht University) - Presenting Author
Professor Martin Neugebauer (Karlsruhe University of Education)
The existence of ethnic discrimination in the hiring process has been repeatedly confirmed using field experiments that confront real employers with fictitious applicants. However, the conditions that exacerbate or alleviate hiring discrimination have received surprisingly little attention by researchers. One reason for this paucity of research is that field experiments do usually not contain the necessary information to study mechanisms behind discriminatory behavior as they solely collect dichotomous answers from employers next to a few general firm characteristics. For example, the diversity attitudes of recruiting personnel and the openness of organizations towards diversity remain in the dark.
Using a unique combination of different data sources, we, attempt to study these attitudes as a source of ethnic discrimination in hiring. First, using data from a nation-wide field experiment in Germany, we confirm previous research on ethnic discrimination in hiring by showing a hiring gap of 7 percentage points between German and Turkish applicants. Second, we study the relationship between anti-immigrant attitudes of employers and ethnic discrimination. As described above, it is difficult to obtain direct information on employers’ attitudes from field experimental data. Therefore, we use three angles to approximate employer attitudes. First, we look at local election results as a proxy for anti-immigrant attitudes. Second, we determine firms’ commitment to diversity by evaluating texts from job advertisements using automated text analysis. And, third, we apply a survey questionnaire to a subset of our field experiment sample that gives us the opportunity to study recruiter attitudes directly.
Mrs Zaza Zindel (German Centre for Integration and Migration Research) - Presenting Author
Mrs Stefanie Hechler (Freie Universität Berlin)
Mrs Elisabeth Zick (German Centre for Integration and Migration Research)
Mr Long Nguyen (German Centre for Integration and Migration Research)
The reliability of self-reported discrimination experiences has long been debated, as such reports depend on individuals’ awareness, interpretation, and willingness to disclose discriminatory incidents. While self-reports capture personal perceptions, they may not fully reflect the structural racism embedded in societal systems. Complementary methods, such as field experiments, provide objective insights by directly observing discriminatory practices in real-world contexts. Unlike surveys, experiments reveal how discrimination unfolds in practice, bridging the gap between perceived and actual experiences. Combining these approaches offers a comprehensive view of discrimination by aligning personal accounts with observable inequalities.
This study uses a combined approach to examine the relationship between self-reported discrimination and observed discriminatory practices in Germany’s housing market. Drawing on data from the NaDiRa.panel 2024, a probability-based online panel, and a field experiment involving over 10,000 standardized rental applications, we analyze racial discrimination against individuals with non-German names. The experiment manipulated applicant names to signal ethnic backgrounds, providing evidence of differential treatment by landlords.
By integrating self-reported survey data with experimental findings, the study explores the extent to which personal accounts align with structural inequalities in the housing market. This multi-dimensional approach not only advances understanding of racism in Germany but also demonstrates the methodological strength of combining self-reports and field experiments to study discrimination.
Mrs Leonie Fuchs (German Center for Integration and Migration Research (DeZIM) )
Mr Kien Tran (German Center for Integration and Migration Research (DeZIM) ) - Presenting Author
Quantitative empirical research on racism and discrimination typically focuses on analyzing groups and group differences. However, the various possibilities for categorization are discussed intensely and controversially.
Using data from the NaDiRa.panel (n > 20,000), various dimensions of differentiation can be contrasted comparatively. Especially for the Muslim group, we have multiple bases for categorization, such as self-identification, external perception, religious affiliation, and country of origin.
Focusing on several experiences of racism and discrimination, this contribution aims to provide insights into how the choice of categorization influences the visibility of such experiences. In this context, we discuss the overall advantages and disadvantages of the different categorizations. Further, on the basis of descriptive and inferential statistical analyses, we address the following points:
On the one hand, the question arises as to whether racism and discrimination are underestimated when country of origin or religious affiliation are taken into account because certain characteristics relevant to racism (e.g., skin color, ethnicity) may be partially ignored. On the other hand, it is an open question whether the use of self-identification or external perception of identity may lead to an overestimation of experienced racism and discrimination due to potential endogeneity problems (reverse causality). Lastly, this results in a discussion about the differences and convergences between these categories in terms of their underlying relationship in measuring racism and discrimination.
Thus, this study contributes to the predominantly theoretical debate on appropriate categorizations for the statistical identification of racism and discrimination in Germany using empirical comparisons.
Ms Megan A. Hendrich (Ipsos Public Affairs) - Presenting Author
Professor Randall K. Thomas (Ipsos Public Affairs)
With recent social dislocations and profound polarization of many groups, attention to prejudices and discriminatory behaviors has significantly increased. Accurately measuring experiences of perceived discrimination using standardized methods is essential to track changes and identify impacted groups. The measurement of discrimination has often varied based on the type of discrimination, rendering comparisons across types of discrimination difficult, if not impossible. To standardize measurement, we can focus on simple filter questions commonly used to measure discrimination. Filter questions often use either dichotomous response formats (DRF) with yes-no responses or multiple response formats (MRF) with “Select all that apply.” DRF typically yield higher prevalence results than MRF across a wide range of topics, including donations, health conditions, and importance measures. We sought to evaluate differences in the occurrence of discrimination using these response formats. In our first study, we reported the results of an experiment on discrimination measures using a probability-based sample of 3,193 U.S. adults. In our second study, we sought to replicate and extend the results with 3,344 U.S. adults. We randomly assigned respondents to use either a DRF or MRF to indicate the occurrence of discrimination. We also randomly assigned them to consider the occurrence in one of three recall periods (“Ever,” “Past 5 years,” or “Past year”). We asked about three randomly selected specific forms of discrimination from a set of seven types (e.g., racial, gender, age) within eight different contexts where discrimination may have occurred (e.g., employment, education, housing). If a person indicated they experienced any discrimination, we asked how negative of an impact it had on them (measuring salience). While the magnitude of reports differed somewhat between studies, the general pattern of results showed higher reports of discrimination for the DRF than the MRF.
Professor Sören Petermann (Ruhr University Bochum)
Mr Daniel Schubert (Ruhr University Bochum) - Presenting Author
The German General Social Survey (ALLBUS) 2016 includes two vignette-based questions measuring views about foreigners as neighbours. For both questions, there are 13 vignettes representing the surrounding 48 households in the immediate neighbourhood. Each vignette differs in the proportion of foreign households. The proportion of foreigners increases evenly from the first vignette (0%) by approx. 8.3 percentage points in each case up to the 13th vignette (100%). Respondents were asked to indicate all neighbourhood vignettes in which they a) want to live and b) do not want to live. The obvious advantage of such a conceptualisation is to determine individual tipping points, which also allow a buffer zone between favoured and unfavoured areas. However, there are three measurement issues with this. Firstly, there is a high proportion of cases with only one specified neighbourhood vignette. Secondly, the results indicate that the underlying linearity assumption (homogeneous neighbourhoods are preferred, heterogeneous neighbourhoods are rejected) does not apply. Thirdly, there is a small but substantial proportion of cases that both like and dislike individual neighbourhood vignettes, i.e. seems to be inconsistent. We address the first two measurement issues by offering solutions that are based on reasonable assumptions about behaviour in the interview situation and from diversity research. Using the ALLBUS 2016 dataset, this presentation analyses the characteristics of preferred and rejected neighbourhood vignettes, offering insights into how tipping points for discrimination preferences can be identified from the vignette results. This work fills a significant gap, as the extensively collected residential preference data in ALLBUS 2016 remain underexplored so far probably because of these measurement issues.
Mr Kien Tran (DeZIM (German Center for Integration and Migration Research)) - Presenting Author
Mr Max Thom (DeZIM (German Center for Integration and Migration Research))
In order to effectively capture groups vulnerable to racism and discrimination, it is recommended to
increasingly utilize self-identification in surveys. Complementing this, perceived external attribution
should also be considered. Both aspects are understood as dynamic categories, with the advantage
of being sensitive to temporal and spatial fluctuations (cf. Aikins & Supik 2018). However, when
measuring experiences of discrimination and racism, it is crucial to analyze the relationship between
these categorizations. Inconsistencies between them may result in methodological challenges and
biases, which can hinder the accurate assessment of individuals' actual experiences.
For instance, individuals may be affected by racism or discrimination associated with a particular
group to which they are externally attributed but do not self-identify. Conversely, individuals who
self-identify with a group but are not accordingly externally attributed may experience racism or
discrimination differently, or to a different extent.
Using data from the NaDiRa.panel – a large-scale survey conducted in Germany (2022) with around
20,000 participants – this study aims to explore how the alignment between self-identification and
external attribution contributes to the perception of experiences of discrimination and the perceived
reasons for those experiences.
Specifically, we examine whether statistically significant differences exist in perceived experiences
based on whether individuals:
• are both self- and externally allocated into specific groups
• self-identify but are not externally attributed to specific groups
• do not self-identify but are externally attributed to the specific groups
This study highlights the complexities of identity categorization and suggests potential extensions to
the measurement of racism and discrimination in Germany.
Dr Benjamin Schulz (WZB Berlin Social Science Center) - Presenting Author
Ms Josefine Matysiak (WZB Berlin Social Science Center)
Internationally, there are various approaches to measuring racial inequalities in education. In the United States, researchers use race as a common category for this purpose, whereas in Germany, for historical reasons, measuring race is not recognized. Instead, migration background is a well-established category here. Recent scientific debates in Germany on racism and related inequalities, however, argue that migration background does not include a growing proportion of the German population who experience racial ascriptions, while including others who are hardly affected by them. Therefore, a new categorization approach is needed to distinguish between racism and migration-related dimensions. Building on the U.S. discourse on the multidimensionality of race, this article analyzes whether measuring reflected ascriptions, specifically individuals' perceptions of ascriptions as (non-) German, can close this data gap. Using data from the German National Educational Panel Study (NEPS), we estimate OLS models on educational inequality based on reflected ascriptions. Our results show a significant association between being perceived as non-German and lower mathematical and German competencies - even after controlling for migration background. Reflected ascriptions, therefore, deserve more attention when analyzing educational inequalities in Germany and internationally.