All time references are in CEST
Scale design, attitude measurement, and cultural differences |
|
| Session Organiser | Mr Lukas Schick (GESIS Leibniz Institute for the Social Sciences) |
| Time | Friday 21 July, 09:00 - 10:30 |
| Room | U6-08 |
Many researchers use rating scales in their surveys to measure respondents' attitudes. Such rating scales demand much of respondents: Namely to translate a continuous, often uncertain attitude intensity into a standardized, ordinal response format. To support respondents in this, questionnaire designers must make careful choices: For example, the number of response options, the verbalization of the response options, or the scale polarity. Researchers conducting multilingual and multicultural surveys face the additional challenge of ensuring an adequate translation or adaption of their scale.
To improve data quality, the aim and the challenge for researchers is to design scales in such a way that respondents have no difficulties in understanding the scale and understand it equally, thus allowing respondents to answer the question as accurately as possible.
But what impact do different scale designs have on the response behavior and, subsequently, response data quality?
In this session, researchers will share their work on the influence of scale design on response behavior, as well as the implementation of new scales. This includes experiments on the verbalization of response options, the number of response options and the comparability of data in cross-national surveys. The session is of interest to questionnaire designers, especially in cross-national settings and to methodologists interested in the complexities of rating scale design.
Keywords: Scale design, attitude measurement, cultural differences
Mr Lukas Schick (GESIS – Leibniz Institute for the Social Sciences ) - Presenting Author
Dr Dorothée Behr (GESIS – Leibniz Institute for the Social Sciences )
Dr Cornelia Neuert (GESIS – Leibniz Institute for the Social Sciences )
Dr Clemens Lechner (GESIS – Leibniz Institute for the Social Sciences )
A challenge for cross-cultural surveys is to translate fully labeled response scales in a way that ensures comprehensibility and comparability between countries. For an adequate translation, two components need to be transferred: the dimension of the scale (e.g., agree - disagree) and the intensity for which different modifiers can be used (e.g., somewhat, strongly) (Harkness, 2003). When using the agree-disagree scale, large cross-national survey programmes translate or adapt the two components differently. German variants of the ‘classical’ agree-disagree scale, for instance, end up translated as “agree – not agree” or “agree - reject.” There is no empirical evidence of whether the different translations are understood comparably.
The present study tries to fill this gap by answering the question how different translations of “agree – disagree” scales in German affect comparability and response behavior.
To test different versions of the agree-disagree scale in German, we conducted an experiment with respondents of a non-probability online access panel in Germany (n=1008). In this experiment, different translations of the ‘classical’ agree-disagree scale used within existing surveys, such as the European Social Survey (ESS) or the International Social Survey Program (ISSP), were compared. Respondents were randomly assigned to one scale version and answered 15 items on various topics. The scales differed in terms of intensity modifiers or the labeling of the dimension ("agree - not agree" or "agree - reject"). We compare the scales and discuss adequate labeling of “agree – disagree” scales and their comparability.
Dr Jozef Zagrapan (Institute for Sociology of the Slovak Academy of Sciences) - Presenting Author
Research shows that there are numerous factors that may influence the respondent´s decision on which answer to choose when answering a survey. In this study, we focus on a potential problem that has received only limited attention – the familiarity of answers, i.e. how familiar or natural the answers are to the respondent. Previous research shows that the extreme positions on a scale, whose labels are perceived as more intense, are chosen less frequently. However, these extreme positions may "attract" more responses if they are described with more familiar labels, even though their intensity may be considered higher. Also, the differences in responses which are sometimes attributed to cultural differences between countries may be due to the translation of questionnaires in international surveys.
To test whether different labeling influences the answers, we designed a split-ballot experiment in the Slovak language. The split ballot was part of a larger survey, The International Social Survey Programme “Environment” module (N = 1013) when the manipulated questions were added to the original pool of questions. While all the respondents were asked to evaluate how much do they agree with the same statements, we assigned different types of labels of the extreme positions to the random half of them. One group had to choose on a five-point scale with the extreme positions being 1 - “completely agree” and 5 – “do not agree at all”, the second group had the options 1 – “definitely agree” and 5 – “certainly do not agree” (note: approximate translation from Slovak). The positions in between were labeled the same. The answers were analyzed using multinomial regression and the results showed no statistical difference between the groups. While these results need to be further confirmed, our experiment suggests that scales in international questionnaires translated with different synonyms may not produce different results.
Dr Hawal Shamon (Forschungszentrum Juelich) - Presenting Author
Dr Diana Schumann (Forschungszentrum Juelich)
Dr Gerrit Stöckigt (Forschungszentrum Juelich)
From a social psychological based conceptual perspective, individuals can be expected to have positive, neutral, negative, or no attitudes toward an issue. From this angle, it is advisable to offer survey participants response scales that allow them to unambiguously map their true answer on the response scale of an attitude question. Regarding single-item measures, this could be implemented with a bipolar response scale that includes a verbalized neutral middle category in addition to a positive and negative answer scale spectrum; and by offering respondents additionally an exit option that allows them to indicate that they have no attitude toward the issue in question. These conceptual considerations are contrasted with studies that point to the consequences of explicitly offering a verbalized middle option as well as exit option on answer distributions. Both may serve respondents as peripheral cue and, hence, enhance strong satisficing. The purpose of this study is to add to the results of studies, some of which date back to a time when the number of (population) surveys, and thus the familiarity of target persons in dealing with survey items, may have been far less pronounced than nowadays. The present study draws on data from a telephone survey with a split ballot experiment that was conducted in Germany in 2022. Survey participants were recruited following a probabilistic sampling strategy and randomly assigned to one of three conditions that varied regarding the attitude scale used to measure respondents’ attitudes towards various energy sources. Besides the comparison of answer distributions, we examine to what extent the different scale options examined affect correlational analyses.
Professor Mansoor Moaddel (University of Maryland) - Presenting Author
This paper offers a scale to measure religious fundamentalism. The scale is tested among Muslims and Christians in the Middle East. It is also tested among a sample of Jewish respondents in Israel and currently being assessed among a nationally representative sample of Israeli Jews and Israeli Arabs. The scale addresses such challenges in the comparative analysis of the subject as variability of religious fundamentalist movements historically, cross-nationally, and across these religions; differences in the definition of fundamentalism, and etymological ambiguity of the term. Conceptualizing fundamentalism as a cluster of core orientations toward one’s and others’ religion, these orientations are categorized into four components: disciplinarian deity, inerrancy or literalism, religious exclusivity, and religious intolerance. Each component is measured by four survey questions. The 16 items make a single fundamentalism scale. This paper discusses the scale’s validity and then verifies its statistical and predictive validity across the Abrahamic faiths. It then assesses the applicability of the construct in the study of fundamentalism in such other faiths as Buddhism, Brahmanism, and Jainism.