ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Factorial Surveys – Methods and Applications
Session Organisers	Dr Hermann Dülmer (University of Cologne) Professor Stefanie Eifler (Catholic University of Eichstätt-Ingolstadt)
Time	Thursday 17 July, 09:00 - 10:30
Room	Ruppert rood - 0.51

Since vignette-designs (e.g., factorial surveys; scenario techniques) as indirect measurement techniques are very common in the social sciences by now, many different applications can be found. Depending on theoretical and methodological objectives, the applied techniques vary in a broad range and lead to different and sometimes inconsistent results. Due to this diversity, findings on methodological and substantial issues can have different meanings and impacts for further research. This session chooses one way of anticipating the diverse field of factorial surveys and vignette-designs in general, and aims at shedding light on the stage of affairs by discussing recent developments and pooling new findings of projects that try to enrich the discussion. The focus of the session is explicitly broad and all contributions dealing with different analytical strategies, empirical designs or substantial research that make use of factorial surveys or other vignette-designs are welcome. Papers matching one of the following aspects are cordially invited to be part of this session:
• comparison and discussion of design-related questions regarding methodological or substantive aspects,
• new developments in measuring intentions with vignettes,
• theoretical ideas for modelling the relationship between intentions and behavior for further empirical analyses,
• cross validation strategies (new approaches, replications),
• discussion of (dis-)advantages of vignette-designs, validations strategies and/or measures,
• issues of data-collection,
• substantive applications of factorial surveys

Keywords: Factorial Survey, Vignettes

Papers

Integrating Correspondence Experiments and Factorial Survey Experiments to Study Labour Market Discrimination

Professor Giovanni Busetta (University of Messina)
Professor Maria Gabriella Campolo (University of Messina) - Presenting Author
Dr Giovanni Maria Ficarra (University of Messina)
Professor Alessandra Trimarchi (University of Messina)

Hiring discrimination remains a pervasive issue in the European labour market, disproportionately affecting ethnic minorities and women, and contributing to inefficiencies in workforce allocation. This study examines two foundational methodologies—Correspondence Experiments (CEs) and Factorial Survey Experiments (FSEs)—to measure and understand this phenomenon. While CEs utilize fictitious resumes sent to real job postings to detect discrimination, FSEs consist in inquiring employers about fictitious profiles in hypothetical hiring scenarios, allowing researchers to examine decision-making processes in controlled settings.
This study aims at highlighting the strengths of these methods to combine them. CEs offer high external validity by capturing real-world discrimination patterns but are limited in their ability to identify underlying biases, such as taste-based versus statistical discrimination. Conversely, FSEs provide deeper insights into the mechanisms driving discriminatory behaviour, albeit at the cost of reduced realism due to their hypothetical nature and potential social desirability bias.
To address these limitations, we propose a longitudinal mixed-method approach. First, a CE is conducted to establish baseline evidence of discrimination across real job markets. Subsequently, a vignette-based survey is administered to the same employers, designed to explore the cognitive and contextual factors influencing hiring decisions. This two-stage design bridges the information gaps inherent in isolated methodologies, enabling researchers to identify the prevalence, sources, and types of discrimination with greater precision.
Our findings underscore the value of integrating these approaches to generate a comprehensive understanding of hiring discrimination. By leveraging the complementary strengths of CEs and FSEs, this study advances methodological innovation in the field of labour market research and contributes to ongoing efforts to reduce discrimination and inequality in Europe.

AI is Held Morally Responsible for Detrimental Outcomes - But only if it isn’t Trustworthy

Dr Patrick Schenk (University of Lucerne) - Presenting Author
Ms Vanessa A. Müller (University of Lucerne)
Mr Lukas Posselt (University of Lucerne)

Imagine an artificial intelligence (AI) diagnosing a patient. Would you hold AI morally responsible for a mistake leading to a patient’s death? Although philosophers deny this question, research has found that laypeople actually do (Bonnefon et al. 2024; Abend/Posselt/Schenk forthcoming). With autonomous AI producing consequences beyond programmable control, responsibility becomes perplexing. People blame technological systems instead of developers or users (Kneer/Christen 2024). This leads to responsibility gaps. Yet, we know little of the exact conditions under which these gaps emerge.
Using a factorial survey experiment (FSE), we test two hypotheses. First, a violation of normative expectations should result in higher attribution of moral responsibility to AI (H1). People expect medical diagnoses to be correct, for instance. If a diagnosis turns out to be incorrect, people attribute more responsibility to agents producing detrimental outcomes (Knobe/Hitchcock 2009). Yet, this effect depends on an agent’s trustworthiness (Alicke et al. 2011). For trustworthy AI, people are motivated to externalize and therefore discount responsibility (H2).
FSEs are especially suited to test these hypotheses. Unlike simple survey items, vignettes provide situational context. In our vignettes, we vary agent type (AI vs. human), the task (eg., medical diagnosis), normative violations (eg., a mistaken diagnosis), among others (Schenk/Müller/Keiser 2024). Respondents rated the agent’s moral responsibility and trustworthiness after each vignette. We used dual mode administration with a stratified random sample of the Swiss population (n=2703) – in contrast to online samples common to this research area.
Consistent with the hypotheses, norm violation (ie., a mistaken diagnosis) leads to increased attribution of moral responsibility (H1) – but only if AI is not trustworthy (H2). Conversely, highly trustworthy agents are shielded from responsibility claims. These findings advance psychological and sociological theories of responsibility attribution and have timely implications for AI policy.

The Social Foundations of Political Hostility: Disentangling Political and Social Identities through a Factorial Survey Experiment

Ms Nelly Buntfuß (University of Technology Chemnitz) - Presenting Author

Increased hostility between supporters of different parties is often attributed to a rising importance of political identities, but it may be more deeply rooted in social structures. This phenomenon - referred to as social sorting - is increasingly being discussed as a driver of political polarization. This study seeks to expand our understanding of the extent to which negative affect is actually political and to which extent this political hostility has a socio-structural underpinning. In order to discriminate between the relative effects of different political and, in reality, often correlated social characteristics, I conducted a factorial survey experiment among the German resident population (N = 1200) in which party affiliation, issue positions, social class, gender, and region were randomised. By showing respondents more or less “sorted” or stereotypical profiles, this study examines whether profiles with fewer cross-cutting attributes provoke greater negative affect and, in turn, create social distance. The findings will deepen our understanding of how political and social identities interact to shape affective polarization.

A Systematic Review of Gender and Ethnic discrimination in Hiring: Evidence from Factorial Survey Experiments

Professor Giovanni Busetta (University of Messina) - Presenting Author
Professor Maria Gabriella Campolo (University of Messina)
Dr Giovanni Maria Ficarra (University of Messina)
Professor Alessandra Trimarchi (University of Messina)

Hiring discrimination remains a significant obstacle to achieving equitable labour markets across Europe. This issue is particularly pressing as economies face labour shortages at both ends of the skill spectrum, highlighting inefficiencies in labour utilisation and the underrepresentation of women and ethnic minorities in the workforce. Addressing these challenges requires a deeper understanding of the mechanisms underlying hiring discrimination.
This study systematically reviews factorial survey experiments (FSEs) conducted between 2010 and 2024 to analyse gender and ethnic discrimination in hiring. These experiments provide a unique methodological approach by simulating realistic hiring scenarios, enabling researchers to explore biases in a controlled yet flexible manner.
Our review employed a modified Population, Intervention, Comparison, Outcome (PICO) framework to refine research questions and establish clear inclusion and exclusion criteria. Following PRISMA guidelines, we selected 21 FSEs studies, focusing on various European contexts. These studies examined key hiring outcomes such as the likelihood of being hired or invited for an interview, with emphasis on the intersection of gender and ethnicity dimensions. Additionally, two studies innovatively employedexpected wages as proxies for productivity, offering further insights into discriminatory patterns.
Findings reveal consistent disparities. Ethnic minorities and women face significantly lower hiring probabilities compared to majority groups. When gender and ethnicity intersect, compounded effects create substantial barriers to employment. Methodologically, FSEs prove effective in capturing these complex interactions, reinforcing their value in labour market research.
This review not only sheds light on the compounded nature of hiring discrimination but also identifies areas for methodological improvement. It recommends expanding the respondent base to include more diverse groups and assessing discrimination throughout the entire hiring process, including interviews and wage negotiations. By emphasising intersectionality and innovative approaches, this research aims to inform evidence-based policies that foster equity and inclusivity in European labour markets.

Evaluating the Generalizability of Factorial Survey Experiments: A Comparison of Convenience and Real-World Samples

Mr David Strauß (University of Applied Sciences BFI Vienna) - Presenting Author

Factorial survey experiments (FSE) frequently rely on convenience samples consisting of various types of "labor rats" for pragmatic reasons. The indirect method employed in FSEs is renowned for achieving high internal and external validity as well as robust reliability. It is often assumed that "labor rats," compared to "real-world samples," provide more consistent responses due to their familiarity with survey scenarios. Building on this assumption, it can be hypothesized that FSE results derived from "labor rats" exhibit higher between-group R² values and, consequently, better generalizability compared to results from real-world samples.
To test this hypothesis, an FSE was conducted focusing on the degrees of freedom and participation in hybrid work environments and their impact on employee motivation. The experiment utilized both a convenience sample and a real-world sample. A 3x3x2x2x3x3 factorial design was employed, generating 324 vignettes organized into 54 decks of six vignettes each. The convenience sample, recruited via a panel provider, comprised n=1,581 participants. The real-world sample, drawn from three different companies, included n=94 respondents.
A GLS multilevel analysis revealed comparable within-group R² values between the two samples. However, the between-group R² values were negligible in the convenience sample, while substantial effects were observed in the real-world sample. These findings suggest that while "labor rats" are accustomed to survey tasks, this does not necessarily result in more consistent responses. The comparable within-group R² indicates that the vignette design of the FSE is effective, with reliability and internal validity considered high.
Nevertheless, the initial hypothesis must be rejected: The generalizability of results—specifically, the effects at the respondent level—should always be critically evaluated in FSE. Despite this, convenience samples remain a viable data source for FSEs, offering a practical foundation for exploratory research.