All time references are in CEST
Promises and problems of factorial surveys and choice experiments - Methodological challenges and substantive conclusions for survey research |
|
| Session Organisers | Dr Hawal Shamon (Forschungszentrum Jülich) Professor Knut Petzold (Hochschule Zittau/Görlitz) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Experimental designs in surveys, such as factorial surveys and conjoint analyses, have become increasingly established in sociological research in recent years. On the one hand, this family of methods is highly flexible and allows for a tailor-made investigation of sociological questions in different research contexts. On the other hand, it is important to understand the advantages and disadvantages associated with these methods. Recently, there are still relatively few methodological studies on construction principles, analytical procedures and the resulting aspects of data quality and quality of results. The session aims to collect and discuss recent results of relevant methodological studies and methodological developments on vignette experiments.
Submissions that investigate methodological questions regarding a) theoretical and methodological foundations (e.g., models of response behaviour in vignette experiments or related procedures, advantages and disadvantages in the investigation of different sociological concepts, sources of error), b) data collection issues (e.g., strategies for recruiting the respondent sample, combination with alternative data formats, implementation in multi-methods research designs), c) design issues (e.g., aspects of the experimental design, presentation format, survey mode, response scales), d) data analysis issues, e) the examination of context effects (e.g., extreme events), f) validity issues (e.g., construct, internal and/or external validity), or any other methodologically relevant issue for factorial surveys and discrete choice experiments are particularly welcome.
Keywords: survey experiments
Mr Jonatan Möller (HSZG) - Presenting Author
Professor Knut Petzold (HSZG)
Factorial survey experiments (FSE) have become a widely used method in social science research for capturing behavioral intentions through hypothetical scenarios. Despite their extensive use, the predictive validity of FSEs—whether and to what extent reported behavioral intentions align with actual behavior—remains a topic of debate. Few systematic validation studies show ambiguous findings across the exemplary fields of application and provide mixed evidence.
The given talk presents a systematic review of existing research, examining the conditions under which FSEs successfully predict real-world behavior. The review aims to identify key factors that influence the alignment between behavioral intentions and actions. Particular attention is given to the realism and complexity of vignettes, social desirability bias, level equivalence, context factors and surveyed population. In addition, the review will take into consideration theoretical frameworks often employed in FSEs, such as social action theories and theories on decision making, to understand how these models may also help to explain observed gaps between intentions and behavior.
By synthesizing the findings, the review seeks to clarify the social settings and methodological conditions under which FSEs provide valid behavioral predictions in order to contribute to the ongoing methodological debate on factorial surveys. Some guiding considerations will be outlined for researchers, aiming to enhance the predictive validity of their survey instruments. In sum, the presentation contributes to the development of FSEs as a robust tool for studying complex social behaviours.
Mr Daniel Schubert (Ruhr University Bochum) - Presenting Author
Professor Sören Petermann (Ruhr University Bochum)
The German General Social Survey (ALLBUS) 2016 includes two vignette-based questions measuring neighbourhood preferences. For both questions, there are 13 vignettes representing the surrounding 48 households. Each vignette differs in the proportion of foreign households. The proportion of foreigners increases evenly from the first vignette (0%) by approx. 8.3% in each case up to the 13th vignette (100%). Respondents were asked to indicate all neighbourhood vignettes in which they a) want to live and b) do not want to live. The obvious advantage of such a survey is to determine individual tipping points, which also allow a buffer zone between favoured and unfavoured areas. However, there are three measurement issues with this. Firstly, there is a high proportion of cases with only one specified neighbourhood vignette. Secondly, the results indicate that the underlying linearity assumption (homogeneous neighbourhoods are preferred, heterogeneous neighbourhoods are rejected) does not apply. Thirdly, there is a small but substantial proportion of cases that both like and dislike individual neighbourhood vignettes, i.e. are inconsistent. We address the first two measurement issues by offering solutions that are based on reasonable assumptions about behaviour in the interview situation and from diversity research. Using the ALLBUS 2016 dataset, this presentation analyses the characteristics of preferred and rejected neighbourhood vignettes, offering insights into how tipping points can be identified from the vignette results. This work fills a significant gap, as the extensively collected residential preference data in ALLBUS 2016 remain underexplored so far probably because of these measurement issues.
Professor Guido Mehlkop (University of Erfurt)
Dr Robert Neumann (Dresden University of Technology) - Presenting Author
Dr Sebastian Sattler (University of Bielefeld)
Mr Hagen von Hermanni (Dresden University of Technology)
When researchers apply factorial surveys, several decisions about the design
and the means of implementation have to be taken. Next to the number of dimensions and number
of levels within each dimension, choices also involve the number of item questions after each
vignette, construction of vignette texts from first or third person perspective, and answer format.
Crucially, a choice has to be made about the selection of vignettes out of the vignette universe and
how they are arranged in vignette sets. This decision is made conditional to the choice of whether
respondents will rate or answer only one vignette of the vignette universe (so called between
design) or whether each individual reads and rates several vignettes during the course of the
questionnaire (within or fully confounded design).
We developed a 4x4x4 factorial survey design on the perceived probability and severity of privacy
violations, depending on who violates the privacy (credit reporting agency, government agency,
family members, or employers), which information is collected (i.e. on who respondents meet,
diseases, supporting political parties, or where respondents spend time), and the source of the
information (i.e., financial records, browser history, social media, or location data). We conducted
both a fully confounded factorial survey (within) design and a between design in two separate quota-
representative samples in Germany (n=2,960 and n= 1,166, respectively) embedded in an online
survey.
We highlight and discuss the methods-effects that are introduced due to the choice of the design.
Foremost, the directions of the effects remain consistent across the two designs. However, we find
that in the between design the magnitudes of effects are larger for the dependent variable “assessed
severity of the privacy violations”. We discuss the consequences of these effects for the question
whether to choose between or within designs.
Dr Silvia Decadri (University of Milan Bicocca)
Professor Fedra Negri (University of Milan Bicocca)
Dr Gaetano Scaduto (University of Milan Bicocca) - Presenting Author
Despite their potential, Visual Conjoint Experiments (VCEs) remain underutilized in the social and political sciences, with a notable lack of systematic guidance for their design and implementation. This study addresses these gaps by presenting a systematic review of the use of VCEs in these fields, introducing a research protocol designed to be generalizable across subdisciplines, and providing an example of their application in political communication. Following the principles outlined in the PRISMA protocol, the systematic review spans disciplines such as political science, sociology, psychology, and urban studies, incorporating already published articles, conference proceedings, and early-access papers in English indexed in Web of Science, JSTOR, and Scopus (N=46). Only studies explicitly employing VCEs with visual stimuli were included, excluding those relying solely on textual inputs. The review examines the potential of VCEs while identifying key limitations and methodological challenges, including the lack of standardization and the need for cultural adaptation. To address these issues and enhance the practical applicability of VCEs, we propose a structured research protocol. This protocol, demonstrated through a 4-country example on political communication, provides guidance for navigating cultural complexities in visual experimental design and includes a ready-to-adapt R code for scholars interested in replication or willing to build over this design.
Dr Sabine Düval (Deutsches Jugendinstitut München) - Presenting Author
Dr Diana D. Schacht (Deutsches Jugendinstitut München)
Dr Fabian Thiel (Universität Konstanz)
Factorial survey experiments (FSEs) are increasingly implemented in longitudinal studies. While theoretical considerations suggest that FSEs could influence panel attrition and data quality, empirical evidence remains limited. A crucial requirement for FSEs is that respondents can handle the complex task of evaluating multiple dimensions within a single coherent judgment. Within this context, this paper addresses two key research questions: (1) How does the use of FSEs influence panel attrition in longitudinal studies? (2) Does the inclusion of FSEs affect response behavior in subsequent survey waves?
Using data from pairfam (waves 10 [2017/18] and 11 [2018/19]), a German panel study on partnership and family dynamics, we analyze differences in panel attrition and item responses among respondents exposed to FSEs with varying levels of complexity. In wave 10, 4,750 respondents completed the FSEs, and 4,414 participated again in wave 11. Respondents in the low-complexity condition evaluated a brief description of a couple’s division of work, whereas those in medium- and high-complexity conditions assessed more detailed and lengthier scenarios. Item responses in wave 11 were analyzed using a set of standard indicators for measuring gender role attitudes.
Our findings show no significant variations in response rates across the three FSE groups. Differences in response behavior were similarly minimal. For instance, respondents exposed to the most complex FSE expressed somewhat more egalitarian views regarding career-oriented women. A statistically significant variation was observed in attitudes toward child-care division, whereas the most complex FSE correlated with reduced child-care behavior of mothers. We found no variations for other housework-related tasks.
Overall, the study provides initial evidence that incorporating FSEs does not adversely affect panel attrition and data quality. Instead, it highlights the potential of FSEs to enrich data without compromising the integrity of longitudinal surveys.
Professor Magdalena Jelonek (Jagiellonian University / Cracow University of Economics) - Presenting Author
Dr Grzegorz Humenny (Jagiellonian University / IBE)
This presentation provides a practical framework for using synthetic data to validate CBC models, highlighting its potential for optimizing survey design (e.g. attribute levels, sample size, profile complexity, interactions) before deploying it in real-world applications.
The example presented is of a choice-based conjoint (CBC) model designed to assess the utility of various job attributes for university graduates (e.g., pay, company environmental activities), which was designed as part of the BKL project. In the first step, the attributes were cognitively pretested, and then the model was evaluated on synthetic data. Empirical (real) data for the model will be collected in the first quarter of 2024 based on a representative sample of fields of study in Poland.
The findings underscore the role of synthetic data as a cost-effective, scalable, and ethical tool for model testing, particularly in scenarios with limited resources. This approach not only enhances research precision but also provides an accessible framework for early-stage model development. The presentation will also highlight the limitations that come with using synthetic data in CBC, as well as the risks that come with unreflective use of such data (por. Agarwal et al. 2024; Muratza et al. 2023).
Murtaza, H., Ahmed, M., Khan, N. F., Murtaza, G., Zafar, S., & Bano, A. (2023). Synthetic data generation: State of the art in health care domain. Computer Science Review, 48, 100546. https://doi.org/10.1016/j.cosrev.2023.100546
Agarwal, A., Agarwal, A. and Vijaykumar, S. (2024). Synthetic Combinations: A Causal Inference Framework for Combinatorial Interventions, https://arxiv.org/abs/2303.14226
Dr Hawal Shamon (Forschungszentrum Jülich) - Presenting Author
Ms Vanessa Schmieja (Forschungszentrum Jülich)
Dr Stefan Vögele (Forschungszentrum Jülich)
Dr Lisa Hannah Broska (Private)
Discrete Choice Experiments (DCEs) allow the examination of individual preferences in decision-making processes in the context of products, services or policies. For this purpose, respondents are exposed to a series of choice sets that usually describe hypothetical product-, service- or policy alternatives that differ regarding attributes that are specific to the researched products, services or policies. In real life, however, decisions are made by taking the social context into consideration when reflecting on attributes specific to the products, services or policies.
In this study, we examine to what extent and how the social context can be modelled in discrete choice experiments. For this purpose, we draw on a Discrete Choice Experiment on the adoption of renewable technologies that was conducted in Germany in 2021. Participants were randomly assigned to different versions of the DCE, in which we varied the social context for the decision-making process.
Dr Dominik Buttler (Leibniz University Hannover) - Presenting Author
Mr Nick Wessel (Leibniz University Hannover)
Factorial survey experiments (FSE) are widely used to study hiring intentions. Respondents evaluate fictitious job candidates, rating the likelihood of hiring or inviting them for an interview. However, FSEs face criticism for limited external validity, with results varying based on vignette specifications, presentation modes, sample characteristics, etc. Moreover, FSEs are prone to social desirability bias, as respondents may provide socially approved answers.
This study explores the validity of FSEs in hiring intention surveys by addressing key questions:
• Do respondent characteristics—such as recruitment experience, HR education, organizational role, or similarity to candidates—influence vignette evaluations and response quality (measured by response length and score variability)?
• Are there differences in vignette evaluations and response quality between samples obtained from different research panel providers and those recruited via 'traditional' email invitations?
• Can short scales for socially desirable responding effectively control for socially desirable responses?
• Do different response scales (e.g., 0-10 ratings versus pairwise comparisons) influence the tendency toward socially desirable responding?
• How do respondents perceive, interpret, and process information in FSE vignettes?
The study draws on data from a large survey (2,000 respondents, 12,000 vignettes) on hiring discrimination conducted in four institutionally diverse countries: Norway, Germany, Poland, and Romania. Respondents included experienced recruiters (owners, managers, HR professionals). The analysis combines quantitative methods, such as linear and logistic multilevel regression models, with qualitative assessments of vignette responses.
Dr Christoph Zangger (Bern University of Applied Sciences) - Presenting Author
Amidst the climate crisis, new forms of public protest and civil disobedience emerged, such as road blockades or targeting pieces of art in museums (Einhorn and Corrigall-Brown 2023; Hess et al. 2024). Especially road blockades have led to heated reactions from the public and politicians, with some calling for harsh punishments of protestors. Apart from loud voices in the media and online commentators (Niceforo 2024), relatively little is known about how the general public feels about punishing taking part in these new forms of civil disobedience – especially in comparison to other acts and infringements that, by law, could be followed by comparable sentences.
In this paper, I use a factorial survey experiment, implemented in three waves of an incentivized ecological momentary assessment with about 200 randomly selected participants in Switzerland. In each wave, respondents were presented three hypothetical individuals who were found guilty of committing different infringements (vandalism, common assault, possession of a small quantity of narcotics, or interference with road traffic during a climate protest). Additionally, background characteristics (migration and gender through corresponding names, age, financial situation, place of residence), remorse, and criminal record were manipulated in a D-efficient design. Respondents were then asked to decide whether these individuals should pay a fine, get a prison sentence (or no punishment at all), as well as the corresponding amounts or duration of the sentence and if it should be conditional or unconditional. Given the longitudinal nature of the data (three waves, each with three observations per respondent), I then use mixed and nested logit models to analyze people’s sentencing preferences. Initial results show that people would punish taking part in a climate protest similarly as simple acts of vandalism, although there is significant heterogeneity in their response to the committed act as well as to background characteristics.