ESRA 2025 Preliminary Program
All time references are in CEST
Promises and problems of factorial surveys and choice experiments - Methodological challenges and substantive conclusions for survey research 2 |
| Session Organisers |
Dr Hawal Shamon (Forschungszentrum Jülich)
Professor Knut Petzold (Hochschule Zittau/Görlitz)
|
| Time | Tuesday 15 July, 15:30 - 17:00 |
| Room |
Ruppert 042 |
Experimental designs in surveys, such as factorial surveys and conjoint analyses, have become increasingly established in sociological research in recent years. On the one hand, this family of methods is highly flexible and allows for a tailor-made investigation of sociological questions in different research contexts. On the other hand, it is important to understand the advantages and disadvantages associated with these methods. Recently, there are still relatively few methodological studies on construction principles, analytical procedures and the resulting aspects of data quality and quality of results. The session aims to collect and discuss recent results of relevant methodological studies and methodological developments on vignette experiments.
Submissions that investigate methodological questions regarding a) theoretical and methodological foundations (e.g., models of response behaviour in vignette experiments or related procedures, advantages and disadvantages in the investigation of different sociological concepts, sources of error), b) data collection issues (e.g., strategies for recruiting the respondent sample, combination with alternative data formats, implementation in multi-methods research designs), c) design issues (e.g., aspects of the experimental design, presentation format, survey mode, response scales), d) data analysis issues, e) the examination of context effects (e.g., extreme events), f) validity issues (e.g., construct, internal and/or external validity), or any other methodologically relevant issue for factorial surveys and discrete choice experiments are particularly welcome.
Keywords: survey experiments
Papers
Measurement issues in a vignette study on neighbourhood preferences
Mr Daniel Schubert (Ruhr University Bochum) - Presenting Author
Professor Sören Petermann (Ruhr University Bochum)
The German General Social Survey (ALLBUS) 2016 includes two vignette-based questions measuring neighbourhood preferences. For both questions, there are 13 vignettes representing the surrounding 48 households. Each vignette differs in the proportion of foreign households. The proportion of foreigners increases evenly from the first vignette (0%) by approx. 8.3% in each case up to the 13th vignette (100%). Respondents were asked to indicate all neighbourhood vignettes in which they a) want to live and b) do not want to live. The obvious advantage of such a survey is to determine individual tipping points, which also allow a buffer zone between favoured and unfavoured areas. However, there are three measurement issues with this. Firstly, there is a high proportion of cases with only one specified neighbourhood vignette. Secondly, the results indicate that the underlying linearity assumption (homogeneous neighbourhoods are preferred, heterogeneous neighbourhoods are rejected) does not apply. Thirdly, there is a small but substantial proportion of cases that both like and dislike individual neighbourhood vignettes, i.e. are inconsistent. We address the first two measurement issues by offering solutions that are based on reasonable assumptions about behaviour in the interview situation and from diversity research. Using the ALLBUS 2016 dataset, this presentation analyses the characteristics of preferred and rejected neighbourhood vignettes, offering insights into how tipping points can be identified from the vignette results. This work fills a significant gap, as the extensively collected residential preference data in ALLBUS 2016 remain underexplored so far probably because of these measurement issues.
Visual Conjoint Experiments in Social Research: A Comprehensive Review and a Guide for Cross-National Applications
Dr Silvia Decadri (University of Milan Bicocca)
Professor Fedra Negri (University of Milan Bicocca)
Dr Gaetano Scaduto (University of Milan Bicocca) - Presenting Author
Despite their potential, Visual Conjoint Experiments (VCEs) remain underutilized in the social and political sciences, with a notable lack of systematic guidance for their design and implementation. This study addresses these gaps by presenting a systematic review of the use of VCEs in these fields, introducing a research protocol designed to be generalizable across subdisciplines, and providing an example of their application in political communication. Following the principles outlined in the PRISMA protocol, the systematic review spans disciplines such as political science, sociology, psychology, and urban studies, incorporating already published articles, conference proceedings, and early-access papers in English indexed in Web of Science, JSTOR, and Scopus (N=46). Only studies explicitly employing VCEs with visual stimuli were included, excluding those relying solely on textual inputs. The review examines the potential of VCEs while identifying key limitations and methodological challenges, including the lack of standardization and the need for cultural adaptation. To address these issues and enhance the practical applicability of VCEs, we propose a structured research protocol. This protocol, demonstrated through a 4-country example on political communication, provides guidance for navigating cultural complexities in visual experimental design and includes a ready-to-adapt R code for scholars interested in replication or willing to build over this design.
Do Factorial Survey Experiments Affect Panel Integrity? Evidence from a Nationwide Study on Family Dynamics
Dr Sabine Düval (Deutsches Jugendinstitut München) - Presenting Author
Dr Diana D. Schacht (Deutsches Jugendinstitut München)
Dr Fabian Thiel (Universität Konstanz)
Factorial survey experiments (FSEs) are increasingly implemented in longitudinal studies. While theoretical considerations suggest that FSEs could influence panel attrition and data quality, empirical evidence remains limited. A crucial requirement for FSEs is that respondents can handle the complex task of evaluating multiple dimensions within a single coherent judgment. Within this context, this paper addresses two key research questions: (1) How does the use of FSEs influence panel attrition in longitudinal studies? (2) Does the inclusion of FSEs affect response behavior in subsequent survey waves?
Using data from pairfam (waves 10 [2017/18] and 11 [2018/19]), a German panel study on partnership and family dynamics, we analyze differences in panel attrition and item responses among respondents exposed to FSEs with varying levels of complexity. In wave 10, 4,750 respondents completed the FSEs, and 4,414 participated again in wave 11. Respondents in the low-complexity condition evaluated a brief description of a couple’s division of work, whereas those in medium- and high-complexity conditions assessed more detailed and lengthier scenarios. Item responses in wave 11 were analyzed using a set of standard indicators for measuring gender role attitudes.
Our findings show no significant variations in response rates across the three FSE groups. Differences in response behavior were similarly minimal. For instance, respondents exposed to the most complex FSE expressed somewhat more egalitarian views regarding career-oriented women. A statistically significant variation was observed in attitudes toward child-care division, whereas the most complex FSE correlated with reduced child-care behavior of mothers. We found no variations for other housework-related tasks.
Overall, the study provides initial evidence that incorporating FSEs does not adversely affect panel attrition and data quality. Instead, it highlights the potential of FSEs to enrich data without compromising the integrity of longitudinal surveys.
FSE Validity in Assessing Hiring Intentions: Insights from an International Survey on Discrimination in Hiring
Dr Dominik Buttler (Leibniz University Hannover) - Presenting Author
Mr Nick Wessel (Leibniz University Hannover)
Factorial survey experiments (FSE) are widely used to study hiring intentions. Respondents evaluate fictitious job candidates, rating the likelihood of hiring or inviting them for an interview. However, FSEs face criticism for limited external validity, with results varying based on vignette specifications, presentation modes, sample characteristics, etc. Moreover, FSEs are prone to social desirability bias, as respondents may provide socially approved answers.
This study explores the validity of FSEs in hiring intention surveys by addressing key questions:
• Do respondent characteristics—such as recruitment experience, HR education, organizational role, or similarity to candidates—influence vignette evaluations and response quality (measured by response length and score variability)?
• Are there differences in vignette evaluations and response quality between samples obtained from different research panel providers and those recruited via 'traditional' email invitations?
• Can short scales for socially desirable responding effectively control for socially desirable responses?
• Do different response scales (e.g., 0-10 ratings versus pairwise comparisons) influence the tendency toward socially desirable responding?
• How do respondents perceive, interpret, and process information in FSE vignettes?
The study draws on data from a large survey (2,000 respondents, 12,000 vignettes) on hiring discrimination conducted in four institutionally diverse countries: Norway, Germany, Poland, and Romania. Respondents included experienced recruiters (owners, managers, HR professionals). The analysis combines quantitative methods, such as linear and logistic multilevel regression models, with qualitative assessments of vignette responses.
