Factorial Surveys - Methods and Applications 2 |
|
| Session Organisers |
Dr Lena M. Verneuer (University of Bielefeld) Professor Stefanie Eifler (Catholic University of Eichstätt-Ingolstadt) Dr Hermann Dülmer (University of Cologne) |
| Time | Thursday 18th July, 14:00 - 15:30 |
| Room | D30 |
Since vignette-designs (e.g. factorial surveys; scenario techniques) as indirect measurement
techniques are very common in the social sciences by now, many different applications
can be found. Depending on theoretical and methodological objectives, the
applied techniques vary in a broad range and lead to different and sometimes inconsistent
results. Due to this diversity, findings on methodological and substantial issues
can have different meanings and impacts for further research.
This session chooses one way of anticipating the divers field of factorial surveys and
vignette-designs in general and aims at shedding light on the stage of affairs by discussing
recent developments and pooling new findings of projects that try to enrich the
discussion. The focus of the session is explicitly broad and all contributions dealing
with different analytical strategies, empirical designs or substantial research that make
use of factorial surveys or other vignette-designs are welcome. Papers matching one of
the following aspects are cordially invited to be part of this session:
• comparison and discussion of design-related questions regarding methodological
or substantive aspects,
• new developments in measuring intentions with vignettes,
• theoretical ideas for modelling the relationship between intentions and behavior
for further empirical analyses,
• cross validation strategies (new approaches, replications),
• discussion of (dis-)advantages of vignette-designs, validations strategies and/or
measures,
• issues of data-collection,
• substantive applications of factorial surveys
Keywords: Factorial Surveys; Vignette-Designs;
Professor Stefanie Eifler (Catholic University of Eichstätt-Ingolstadt)
Dr Knut Petzold (Ruhr-Universität Bochum)
Ms Hannah Schwarzbach (Catholic University of Eichstätt-Ingolstadt) - Presenting Author
Factorial surveys or vignettes were frequently used in the analysis of norm-related issues. However, up to now it remained an open question whether the presentation mode of vignettes plays an important role for measuring judgements. The study is based on the assumption that the presentation mode of vignettes – either verbal or visual – will have an impact on the assessment of judgements concerning feelings of safety in public places. We conducted a split-ballot experiment in order to study differences in judgements according to presentation mode. The factorial survey was realized based on a within-subjects design. The vignettes refer to situations in public places, characterized by the presence or absence of physical disorder, social disorder, passers-by, video-surveillance, and light, and the level of wideness of place. A web-based survey of adults aged 17-60 was conducted (n=1.449, nverbal=714, nvisual=735). Fixed-effects multilevel-analyses revealed the result that effect differences of safety reports emerge for physical disorder, passers-by and light. Accordingly, features of the vignettes interact with the respective presentation mode. The results are discussed with regard to methodological considerations.
Dr Lena M. Verneuer (University of Bielefeld) - Presenting Author
Professor Stefanie Eifler (Catholic University Eichstätt-Ingolstadt)
Mrs Hannah Bucher (GESIS – Leibniz- Institut für Sozialwissenschaften)
Sensitive questions in surveys can lead to systematic under- or overreporting of desirable or undesirable behaviors or attitudes. This phenomenon is referred to as social desirability, e.g. the tendency to not answer truthfully according to social norms.
In the context of surveying for exmaple criminal behavior, people might not answer truthfully according to external social norms (that are grounded in the criminal law)
and underreport different deviant behaviors. One way of anticipating this problem (not only in criminal research) is to implement indirect techniques, such as randomized response techniques or vignettes, that broaden the distance between respondent and the behavior/attitudes in question. The main aim of vignette-designs in general is to present an every-day-situation to which respondents should easily and truthfully give an answer.
What is not surveyed yet, is the quesion, which effect the first- and the third-person wording in a situational description has on the extend of truthfull answers respectively the extend of social desirability: Finch (1978:110) argues that a third-person wording leads to more valid answers because respondents might not feel like presenting their own behavior/attitudes which in turns leads to more valid answers. At the same time, it is possible to argue that persons might feel a higher cognitive burden to associate themselves with a third person and thus answer less adequate (and maybe less truthfully, too).
The talk will focus on this unanswered question by looking at both arguments in more detail: Besides possible explanations and effects of both arguments,
results of an experimental design with individual- and other-based vignettes (conducted in the context of the study "Mode Effects in the Surveying of Sensitive Items (MESSI)") will be analyzed in reference to their effect on the respondents' social desirability.
Dr Hawal Shamon (Forschungszentrum Jülich, Institute of Energy and Climate Research Systems Analysis and Technology Evaluation (IEK-STE)) - Presenting Author
Professor Hermann Dülmer (University of Cologne, Institute of Sociology and Social Psychology (ISS))
The factorial survey is an experimental design in which the researcher constructs varying descriptions of situations or individual persons (vignettes) which will be judged by respondents with regard to a particular aspect. Some researchers present vignettes in text format as short stories, others present the central information of vignettes in a tabular format. To date only a few sentences have been published, by Auspurg and Hinz (2015), on the impact of the presentation format (text vs. table) on the answer behavior of students. Empirically, no differences were found between either format. We discuss a cognitive scheme for the question-answer-process in factorial surveys and define indicators for weak and strong satisficing in factorial surveys. In order to examine our hypotheses, we conducted an internet experiment on the basis of a quota sample in Germany.
Professor Christiane Gross (University of Wuerzburg) - Presenting Author
Using factorial survey (FS) design has become more and more popular in recent years. The advantages are obvious: Firstly, FS is able to detect causal effects due to its experimental character. Secondly, FS can easily be combined with a common questionnaire, allowing the estimation of effects on vignette, and respondent level as well as and cross-level effects between vignette and respondent level. However, previous research on FS also revealed some shortcomings: So far, the respondent’s social context has been ignored and knowledge about the external validity of the vignette judgements (outcome) is inadequate.
This contribution will address the prior shortcomings by modeling social context within sets. We exemplify this approach with educational research on early tracking decisions of teachers, namely the research question: Based on what characteristics do elementary teachers in Germany recommend children to ability tracks (“Gymnasium” or “Hauptschule”) during the last year of elementary school?
We model the social context of these key recommendations by creating (at least) two different sets of vignettes. One set of vignettes includes e.g. 20 vignettes that represent students of class 4a with very good grades on average and one of 20 vignettes that represent students of class 4b with poorer grades. By using this design, we could examine the so-called “frog-pond effect”, which refers to the scenario of a person possibly benefitting from “bad” social context when being evaluated within a group.
This approach of systematically manipulating the mean value of grades as one vignette dimension of fictive students can easily be transferred to other fields of research beyond educational decisions, whenever the fictive vignettes build a certain group. One can also apply this approach with categorical variables by systematically manipulating the frequencies of the vignette dimension e.g. to show a certain gender ratio in teams.