All time references are in CEST
Developing Innovative and Model-Based Interventions to Combat Survey Satisficing |
|
| Session Organisers | Ms Julia Witton (German Institute for Economic Research) Dr Carina Cornesse (Free University of Berlin and German Institute for Economic Research) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
The concept of satisficing was introduced to the survey context by Krosnick in 1991. Since then, it has kept researchers busy and remains one of the most important contributors to total survey error. Respondents exhibit problematic response behaviors such as midpoint or endpoint selection, nondifferentiation, or item nonresponse, aiming to reduce the cognitive burden that the survey response process entails. These response strategies evidently impair the quality of the generated data, which in turn affects the validity and reliability of the findings that researchers derive from this data. In times of increasing financial costs and decreasing response rates, it is particularly necessary to maximize quality when collecting survey data. Numerous theoretical insights were gained in the past decades, and there are some general recommendations to prevent participants from satisficing. While in several domains of survey data collection, innovative interventions to maximize the collected data’s quality (e.g., targeted data collection mode assignment) have become increasingly relevant, innovation on measures that address the prevention of satisficing is still in its infancy. There are, in fact, some promising results with targeted interventions such as prompting participants when speeding is detected or when they try to skip a question before answering it.
In this session, we want to shed light on current developments in the field that help to innovate the survey landscape’s handling of satisficing. We aim to put a spotlight on the practical aspects and welcome contributions that address existing challenges of satisficing in longitudinal as well as cross-sectional data collection settings. We are particularly interested in the following contributions:
- Interventions based on statistical models (e.g. latent class analyses or prediction models)
- Experimental assessments of satisficing interventions (e.g. split-ballots)
- Longitudinal approaches to measuring and combating satisficing
Keywords: satisficing, survey data collection, measurement error, interventions
Dr Laura Fumagalli (ISER, university of essex) - Presenting Author
Dr Elena Fumagalli (Utrecht University)
Information about people's social attitudes is often collected using Likert scales. However, respondents sometimes try to minimize the survey response burden by giving "spurious neutral answers," selecting the midpoint of the Likert scale as a more socially acceptable alternative to non-response. This behavior can affect data quality and the conclusions drawn from the data. For example, women and men often report different social attitudes, with women's reported attitudes typically being less extreme. If women are more prone to midpoint selection, the Likert scale may capture a gender difference in response behavior, rather than actual attitudes.
We test this hypothesis by analyzing attitudes towards nuclear energy from the Understanding Society Innovation Panel. To identify spurious neutral answers, we use a split-ballot experiment inducing random variation in the availability of non-response as an alternative to midpoint selection, along with a Middle Inflated Ordered Probit (MIOP) that jointly models midpoint selection behaviour and social attitudes, leveraging the variation induced by the experiment to construct exclusion restrictions. Our findings indicate that women report more neutral answers than men, due to a higher propensity for giving spurious neutral answers. There are no gender differences in reporting genuine neutral attitudes. We test the performance of the MIOP with a widely available non-experimental exclusion restriction. Results from both experimental and non-experimental exclusion restrictions are very similar. This suggests that the MIOP can be used for other outcomes and surveys where experiments are not available.
Finally, we use the MIOP to analyze attitude questions from Understanding Society. Women are significantly more likely than men to use spurious neutral answers when reporting attitudes on stereotypically male topics, such as political issues; men are slightly more likely to use spurious neutral answers in attitude questions on stereotypically female topics, such as family issues.
Mr Lukas Olbrich (IAB Nuremberg) - Presenting Author
Professor Joseph W. Sakshaug (IAB Nuremberg, LMU Munich, University of Mannheim)
Professor Eric Lewandowski (NYU)
Inattentive respondents pose a substantial threat to data quality in web surveys. To minimize this threat, we evaluate methods for preventing and detecting inattentive responding and investigate its impacts on substantive research. First, we test the effect of asking respondents to commit to providing high-quality responses at the beginning of the survey on various data quality measures. Second, we compare the proportion of flagged respondents for two versions of an attention check item instructing them to select a specific response vs. leaving the item blank. Third, we propose a timestamp-based cluster analysis approach that identifies clusters of respondents who exhibit different speeding behaviors. Lastly, we investigate the impact of inattentive respondents on univariate, regression, and experimental analyses. Our findings show that the commitment pledge had no effect on the data quality measures. Instructing respondents to leave the item blank instead of providing a specific response significantly increased the rate of flagged respondents (by 16.8 percentage points). The timestamp-based clustering approach efficiently identified clusters of likely inattentive respondents and outperformed a related method, while providing additional insights on speeding behavior throughout the questionnaire. Lastly, we show that inattentive respondents can have substantial impacts on substantive analyses.
Ms Julia Witton (German Institute for Economic Research (DIW Berlin)) - Presenting Author
Dr Carina Cornesse (GESIS - Leibniz Institute for the Social Sciences)
This study investigates the potential of targeted interventions to prevent satisficing in survey research. The presentation focuses on the robustness and predictiveness of survey satisficing in self-administered mixed-mode panels. The key research interests include identifying distinct patterns of satisficing behavior using latent class analysis (LCA), replicating these patterns across survey waves and modes, determining respondent characteristics that correlate with identified satisficing patterns, and predicting future satisficing behavior based on previous survey waves. The analysis was based on the first three waves of the German Social Cohesion Panel (SCP, jointly conducted by the German Institiute for Economic Research, DIW Berlin, and the Research Institute Social Cohesion, RISC), a mixed-mode panel survey with participants self-selecting into either paper-and-pencil (PAPI) or web (CAWI) mode. The study identified three latent classes that replicate over all waves in CAWI and over the first two waves in PAPI mode: 1) Optimizers, who exhibit the least and unspecific satisficing behavior, 2) ExtreMists, who skip questions and select the endpoint of response scales, 3) Indifferents, who select the scales’ midpoints and are at risk of speeding through the survey. Using multinomial regression analyses, we find individual-level characteristics such as education and income to be associated with the latent satisficing pattern with limited predictive power. Binomial logistic regression analyses reveal that the estimated posterior probabilities of belonging to a satisficing class in a previous wave were consistently significant predictors of belonging to that same class in the future. In conclusion, the results emphasize the need for innovative, targeted interventions considering situational factors. Given the robustness of identified patterns, tailored intervention strategies should be developed to effectively combat satisficing. Our research suggests that mode-specific approaches for such interventions may be necessary.