All time references are in CEST
Sensitive Questions and Social Desirability: Theory and Methods 2 |
|
| Session Organisers | Dr Felix Wolter (University of Konstanz) Professor Jochen Mayerl (Chemnitz University of Technology) Dr Henrik K. Andersen (Chemnitz University of Technology) Dr Justus Junkermann (University of Leipzig) |
| Time | Wednesday 19 July, 16:00 - 17:30 |
| Room | U6-21 |
Misreporting to sensitive survey questions is an age-old and notorious problem in survey methodology. Empirical evidence has shown that survey respondents tend to engage in self-protective behavior (e.g., by not answering truthfully) when it comes to questions on private issues, deviant behavior, or attitudes that do not conform with social norms (e.g. sex, health, income, illicit drug use, tax evasion or xenophobia). This leads to biased estimates and poor data quality. Although a large body of methodological literature addressing these issues does exist, many questions remain. Further, recent literature has called special questioning techniques such as the crosswise model or other randomized response techniques into question, for instance because they have been accused of generating false-positive bias.
This session aims at deepening our theoretical and practical knowledge with respect to why and when response- or social desirability bias occurs and how best to avoid it. In particular, we are interested in studies that (1) focus on explaining the psychological mechanisms that lead to misreporting on sensitive survey questions; (2) present conceptual or empirical research on special questioning techniques (e.g., randomized response and item count techniques, factorial surveys) aimed at mitigating response bias; (3) deal with statistical procedures to analyze data generated with special data collection methods; (4) address other topics related to social desirability and/or response bias, e.g., mixed methods of data collection, using “big data” and/or record linkage techniques, but also research ethics and data protection issues.
Keywords: Sensitive questions, social desirability, response bias, data collection techniques
Professor Pier Francesco Perri (University of Calabria) - Presenting Author
Dr Adrian Hoffmann (Heinrich Heine University Düsseldorf)
Dr Shu-Hui Hsieh (Research Center for Humanities and Social Sciences, Academia Sinica )
Behavioral recommendations, restrictions, and protective measures issued by governments and public health agencies to combat the COVID-19 pandemic have had a significant impact on the daily behavior of people around the world. To evaluate the effectiveness of such interventions, it is of utmost importance to obtain valid prevalence estimates for their adherence. However, when asked about COVID-19-related behaviors with conventional direct questions (DQ), some respondents may prefer to protect their privacy rather than answer truthfully; this may lead to nonresponse, or underreporting of socially undesirable and overreporting of socially desirable behaviors. Indirect questioning techniques, such as the recently proposed Cheating Detection Triangular Model (CDTRM), aim at controlling for nonresponse and social desirability bias by guaranteeing the confidentiality of individual answers. The CDTRM has the additional advantage of providing lower and upper bounds for the estimated prevalence, taking participants’ instruction nonadherence into account. Using the probability-based online panel of Taiwanese adults surveyed by the Centre for Survey Research (Academia Sinica) between October and November 2022, we experimentally compared prevalence estimates obtained in a CDTRM (n = 1,144) and a conventional DQ condition (n = 573) for three COVID-19-related behaviors with different levels of sensitivity. For a nonsensitive behavior (having received a COVID-19 vaccine), prevalence estimates were expectedly comparable between experimental conditions (CDTRM: 87-93%, DQ: 92%) and consistent with official vaccination data in Taiwan (95%). For a behavior with low sensitivity (having been tested positive for COVID-19; CDTRM: 42-56%, DQ: 38%), and especially for a highly sensitive behavior (having concealed a positive COVID-19 test result from others; CDTRM: 18-36%, DQ: 5%), estimates in the CDTRM condition were substantially higher, thus presumably less distorted, and more valid, than in the DQ condition. Our results, therefore, suggest the CDTRM as a promising method
Ms Larissa Fritsch (University of Zurich) - Presenting Author
Dr Sandra Gilgen (University of Zurich)
Ms Maila Mertens (University of Zurich)
Professor Jörg Roessel (University of Zurich)
The last decades have seen enormous innovations in artificial reproduction technologies (ART), such as sperm donation or IVF. However, this has led to controversial public debates on who should have access to and receive financial support for the use of these new technologies. The main point of controversy is whether access should be limited to (married) couples, and whether same-sex couples should be eligible. Empirical insights on this social justice issue feeding into existing inequalities regarding reproductive self-determination are scarce and much needed.
Using two methods to measure the degree of support for access to ART, we aim to make both a substantive and methodological contribution. One half of the respondents – in a nationally representative survey that will be conducted in Switzerland in early 2023 – are shown two factorial survey experiments on normative and financial support for ART while the other half are asked direct questions measuring their support.
Implementing the survey experiments allows us to measure the independent effects of multiple treatments (e.g., whether the described people are single or coupled; hetero- or homosexual; already have children or not; their age and social background) influencing people’s assessments regarding access to ART. We hypothesize that these assessments, essential to questions of reproductive justice, are shaped by the social structural position of respondents, as well as their broader gender role as well as heteronormativity attitudes.
Comparing the results from the experiments with answers to the direct questions regarding some of the subgroups allows us to assess the magnitude of the resulting discrepancy. This is especially valuable in cases pertaining to controversial topics such as ART support, or more broadly reproductive justice, that are particularly subject to social desirability bias.
Dr David Randahl (Uppsala University)
Dr Sophia Hatz (Uppsala University) - Presenting Author
Researchers seeking to measure and understand public opinion on controversial topics face a fundamental problem: survey respondents tend to report ‘socially desirable’ opinions and values, while concealing their true attitudes. In the aggregate, this translates into systematically biased measures of public opinion: undesirable views are undercounted. We make an important contribution by demonstrating that social desirability bias can cut in opposite directions across population subgroups. For example, what is social desirable to Republicans may be socially undesirable to Democrats, leading Republicans to over-report and Democrats to under-report. We use list experiments embedded in public opinion surveys to demonstrate that subgroups defined by political affiliation and race misreport sensitive opinions in opposite directions. We show that when this is the case, the common list experiment survey technique does not return accurate measures of sensitive topics. The findings have important implications for political science and survey research: public opinion may not be as polarised along subgroup-lines as current survey methodologies lead us to think.
Dr Keith Smith (ETH Zurich)
Mr Florian Lichtin (ETH Zurich) - Presenting Author
Ms Franziska Quoss (ETH Zurich)
Ms Sarah Gomm (ETH Zurich)
Item count techniques (ICT) survey embedded experiments are commonly implemented investigate social desirability biases in survey responses (Imai 2011). Typically, ICT has been adopted to identify embarrassing, or even illegal, behaviors or attitudes (Creighton et al. 2019), yet social desirability is normative, and relative to the respondent’s social identities. That is, what is deviant for social identity (such as an environmental activist eating a McDonald’s hamburger), may be normative for another. And people with certain social identities may be more likely to be untruthful in their responses. Using ICT experiments across three surveys in Switzerland and the United States, we investigate how responding to voluntary engagement in pro-environmental behaviors (e.g. purchasing an electric vehicle, installing household solar panels, avoiding short-haul flights) varies by individual characteristics (e.g. environmental attitudes, political orientations, socio-demographic characteristics). These analyses contribute to emerging research into over-reporting of voluntary environmental behaviors, the role of social identities shaping environmental behavioral intentions and survey embedded experimental designs.
Mr Tlegen Kuandykov (Central Asia Barometer) - Presenting Author
The study of public opinion in Central Asia is a rather specific activity, requiring ingenuity and caution at all stages of research due to the authoritarian context, which greatly expands the field of 'sensitive' questions and creates serious methodological concerns in the measurement of public opinion.
If it is safe to survey public opinion about media, culture and the environment, and it will not compromise the results of the poll, other specific topics, such as questions about religion in Tajikistan, or central government and president in Uzbekistan, pose a serious methodological problem. In most cases, respondents either evade answering or misreport, presenting as their opinion socially desirable answers and answers that coincide with the official position of the authorities or, as perceived by the respondent, do not contradict it.
In this paper I intend to present the empirical findings from the Central Asian Barometer experience of polling, as well as the challenges and possible solutions to it. In particular, I intend to reflect on three issues: 1) the systematic bias concern present in authoritarian contexts. 2) High political, religious and ethical sensitivity 3) Security concerns both of the respondent and the interviewer due to topic sensitivity. One of the most effective responses to these constraints used by the CAB is to develop specific question wording that helps to alleviate some of the constraints and reduce the sensitivity.