All time references are in CEST
Frenemies: Understanding and interpreting the impact of answer scale orientation on survey data quality |
|
| Session Organisers | Dr Yongwei Yang (Google) Dr Mario Callegaro (Google) Mr Aaron Sedley (Google) |
| Time | Thursday 20 July, 09:00 - 10:30 |
| Room | U6-01b |
Survey researchers are wary of answer scale design choices’ impact on data quality and respondent experience. One important area of work surrounds scale orientation with visually-displayed rating scale questions. Answer scale orientation may be described in two ways: display orientation (vertical or horizontal) and positive pole location (top or bottom for vertically orientated scales; left or right for horizontally orientated ones). Important theories about how answer scale orientation may affect survey response have been offered alongside empirical evidence. These include primacy effect due to satisficing (Krosnick & Alwin, 1987; Chan, 1991; Garbansky, Schaeffer, & Dykema, 2019; Mavletova, 2013; etc.), primacy effect due to anchor-and-adjust (Yan & Keusch, 2015), and interpretation heuristics (Tourangeau, Couper, & Conrad, 2004, 2013). The applicability of these theories in 3MC context has also been explored (Ferrall-Nunge & Cooper, 2011; Yang, Timpone, Callegaro, Hirschorn, Achimescu, & Natchez; etc.). Nevertheless, this body of work remains limited and there is a need for further testing the generalizability of competing theories in diverse contexts and for clearer and practical recommendations. This session will facilitate the expansion of this important area of work. Specifically, we seek contributions that systematically evaluate the generalizability of existing or new theories about answer scale orientation along various dimensions, such as:
-- Different substantive constructs (e.g., customer satisfaction, user trust, political attitude)
-- Construct polarity (bipolar versus unipolar)
-- Using verbal labels versus emojis
-- Full versus partial labeling
-- Language and cultural settings
For quantitative studies, we encourage contributions that use high quality samples. We also encourage contributions that dig deeper into the response mechanisms using qualitative or observational techniques.
Finally, we encourage works that carefully investigate existing evidence through systematic reviews or meta-analysis.
Keywords: question and questionnaire design, answer scale, data quality, validity, psychology of survey responses, cross-cultural, primacy effect, satisficing, interpretation heuristics, bipolar and unipolar, verbal labels and emojis
Dr Martin Pielot (Google)
Dr Mario Callegaro (Google Cloud) - Presenting Author
Customer satisfaction surveys are common in technology companies like Google. The standard satisfaction question asks respondents to rate how satisfied or dissatisfied they are with a product or service generally going from very satisfied to very dissatisfied.
When the scale is presented vertically, some survey literature suggests placing the positive end of the scale on top as “up means good” to avoid confusing respondents.
We report from 3 studies. The first study shows that reversing the response options of a bipolar satisfaction question (very dissatisfied on top) leads to significantly lower reported satisfaction. In a between group experiment, 3,000 Google Opinion Rewards (Smartphone panel) respondents took a 1-question satisfaction survey. When the response options were reversed participants were 10 times more likely to select the very dissatisfied option (5% versus 0.5% prevalence). They also took 11% more time to answer the reversed scale.
The second study shows that this effect can be partially explained by respondents mistaking the word dissatisfied for satisfied. ~1750 people responded to a reversed satisfaction question in an in-product survey on fonts.google.com. In a follow-up verification question (“You selected [answer option], was this your intention?”), 42.1% of the respondents indicated that they had selected very dissatisfied by mistake. Open ended feedback suggests that respondents hadn’t read properly and expected the positive option on top.
The third study shows that this effect does not occur with a unipolar satisfaction question (not at all satisfied, …, extremely satisfied). In a replication design of the first study, 3,000 Google Opinion Rewards (smartphone panel) respondents took a 1-question satisfaction survey. A unipolar satisfaction question was used and reversed 50% of the time. The 10x increase in dissatisfaction from Study #1 was not replicated. There was no significant difference in the number of low-satisfaction responses (not at all satisfied) but the percent of extremely satisfied was 29% when it was on top and 39% when it was at the bottom.
These findings seem to suggest that reversing a scale from positive to negative and negative to positive have different effects depending on the scale (bipolar vs unipolar). For example, in the last experiment the primacy effect was not found but the opposite.
More experiments should be conducted on different samples to better understand the interaction of scale orientation versus the type of scale (unipolar vs. bipolar).
Ms Fabienne Wöhner (University of Bern) - Presenting Author
Professor Axel Franzen (University of Bern)
Satisfaction with life is one of the key concepts in the social sciences. In national and international surveys, life satisfaction is often measured with a single item on an 11-point answer scale. The aim of this study is to gain further insights into the reliability and validity of the life satisfaction item regarding the positive pole location and the display orientation (horizontal vs. vertical) of the response scale. Previous research suggests that the positive pole location should not change the average or standard deviation of the construct measured. Accordingly, this hypothesis is tested using an experimental design. We manipulated the answer scale so that it either starts or ends with the positive pole (“completely satisfied”). The differently directed answering scales were randomly assigned to the respondents. Furthermore, a significant share of respondents decided to answer the online questionnaires on their smartphones. Usually, longer horizontal scales, such as the 11-point answer scale, cannot be represented adequately on mobile devices. Hence, these respondents see a vertical version of the scale. Accordingly, the second topic of this study is to compare the results obtained when using different scale orientations. As respondents can decide on the device type by themselves, multivariate models are applied to control for possible selection effects. Data from a university-wide online student survey at the University of Bern in Switzerland are used (N = 3150). Results show that the average life satisfaction differs according to scale direction (mean diff = 0.7, p < 0.001). No differences were found concerning horizontal and vertical orientation. The results suggest that scales can be optimized for mobile devices without severe issues, as horizontal and vertical scales lead to similar conclusions. However, the order of the answering categories needs to be chosen carefully, as this might affect the measurement.
Ms Freja Wessman (SOM Institute) - Presenting Author
Dr Sebastian Lundmark (SOM Institute)
Mr Felix Cassel (SOM Institute)
A general assumption in survey methodology research is that respondents follow a set of simple heuristics when interpreting the meaning of questions and that respondents expect questionnaires to follow conversational norms. The focus of this study is to investigate the conversational norm called “Up Means Good”, where respondents are believed to interpret the topmost response option in a vertical list as the most desirable option (Tourangeau et al., 2004) and that respondents expect the topmost option to be the most positive or greatest value and the bottommost option be the most negative or smallest in value. In addition, the present study also assesses whether respondents expect horizontally oriented lists to follow a conversational norm of “Left Means Good”. That is, whether respondents expect horizontally ordered lists to start with the most positive or highest intensity option to the left and the most negative or lowest intensity option to the right. Deviating from these conversational norms is expected to result in respondents’ taking longer time to answer the questions and delivering data of worse quality. These expectations are assessed in an experiment administered to respondents in the Swedish Citizen Panel, where respondents were randomly assigned to answer questions with either response options presented vertically or horizontally. Furthermore, respondents were randomly assigned to see the response options with either the highest intensity option at the top or at the bottom (for vertically oriented response options) or as the leftmost or rightmost option (for the horizontally oriented response options). The four versions of the response options were assessed by self-reported satisfaction with the questions, and data quality in terms of concurrent validity and item non-response.
Dr Femke De Keulenaer (Ipsos EPA) - Presenting Author
Dr Joke Depraetere (Ipsos EPA)
Ms Cristina Tudose (Ipsos EPA)
Ms Christine Tresignie (Ipsos EPA)
Using data from KnowledgePanel Europe, new empirical evidence is provided on the relation between survey response and scale orientation. KnowledgePanel is a large-scale random probability online panel and has been operating in the US since 1999 and in the UK since 2020. Building on this work, we are now expanding the offering across Europe. Within KnowledgePanel Europe, a multi-country design is applied in which similar profiling variables are used during recruitment of panel members. This uniform design allows to make cross-country comparisons based on these profiling variables.
Many of the studies investigating the impact of answer scale orientation, focused primarily on a single country. Potential cultural and language effects are, therefore, not considered. In this session, we present the results of various split-ballot designs conducted within KnowledgePanel. The split-ballot designs randomly allocated respondents into two groups that were shown reversed response scales. The strength of this design lies in the construction of independent random samples from a random probability population panel with the possibility of making cross-national comparisons. Results are presented across four countries (Italy, France, Poland and Sweden), and across various survey topics and question types. The uniform profiling approach allows to make additional comparisons among socio-demographic and attitudinal data at cross-national level.
In this presentation, we discuss overall question order bias in an online survey setting. Emphasis is placed on which survey topic and types of survey questions show the largest bias. The impact of cultural and language are assessed and comparisons are made across socio-demographic and attitudinal characteristics.