Researching Sensitive Topics: Improving Theory and Survey Design 4 |
|
| Chair | Dr Ivar Krumpal (University of Leipzig ) |
| Coordinator 1 | Professor Ben Jann (University of Bern) |
| Coordinator 2 | Professor Mark Trappmann (IAB Nürnberg) |
In survey research, social desirability bias (SDB) describes non-sampling error in survey data due to respondents’ tendency to provide answers that present them in a positive way to others, because they think the reported attitudes or behaviours are socially more acceptable. SDB is particularly a problem for questions asking about sensitive topics. In higher education and science research we may expect socially desirable response patterns on topics such as academic misconduct, but also about the demographic background of staff and students (e.g., ethnicity, gender), the relationship between supervisors and supervisees, or substance abuse among staff in stress situations.
In order to reduce SDB in survey responses, survey methodologists continuously develop techniques to improve respondents’ anonymity and confidentially when asking questions. One promising approach is the so-called Crosswise Model (e.g., Yu et al. 2008; Krumpal et al. 2015), which has proved successful for questions about substance abuse among athletes, plagiarism, and tax fraud (Coums et al. 2011; Jann et al. 2012; Höglinger et al. 2014; Shamsipour et al. 2014). A comprehensive evaluation of the applicability of the model, especially in the area of higher education and science study, is still open to academic research.
This paper investigates different questions in this area and evaluates the respondents’ cognitive understanding of the Crosswise Model. Systematic analysis their respondents reveals how far respondents feel that their privacy and anonymity is protected by this question format and whether some groups are more suspicious than other that this is an unusual question format. The analysis is based on systematically conducted cognitive interviews about the method among respondents at different educational levels in Germany and the United Kingdom.
The valid measurement of sensitive issues such as norm-violations or stigmatizing traits through self-reports in surveys is often problematic. Special sensitive question techniques such as the Randomized Response Technique (RRT, Warner 1965) and among its variants the recent Crosswise Model RRT (CM, Yu, Tian and Tang 2008) should generate more honest answers by providing full response privacy. Also, the Item-Count Technique (Miller 1984), which is based on a similar logic, is increasingly employed in research on sensitive topics. However, these methods sometimes do not work as expected and evaluating whether particular implementations actually improve data validity is essential before their application in substantive surveys. To this end, studies so far mostly compared prevalence estimates of sensitive traits and behaviors of different techniques. Assuming that respondents only falsely deny but never falsely admit a sensitive trait or behavior, higher prevalence estimates were interpreted as more valid estimates. However, if false positives occur, i.e., if respondents are misclassified as bearing a sensitive trait although they actually do not, conclusions drawn under this assumption are likely wrong. False positives occurred knowingly in one variant of the Crosswise Model RRT. Other RRT variants might be affected too, but studies so far have largely neglected this possibility and did not test for it. We show an evaluation approach that detects systematic false positives without the need of a validation criterion – which is often unavailable. Results from an application in a survey on “Organ donation and health” show, that the Crosswise Model RRT produced false positives to a non-ignorable extent. This finding is in line with a previous validation study, but has not been revealed by several comparative evaluation studies that did not consider false positives. We will test for false positive responses with an extended replication of the survey including various RRT and the Item-Count Technique.
The idea of reducing nonresponse and untruthful answering by applying an alternative questioning design that protects the respondent’s privacy was originally presented for binary variables. In the first effort to apply such a method also to quantitative sensitive variables (such as the number of abortions per unit in a given population of women) it was suggested to ask a respondent either with a certain probability for the true value of the interesting variable or with the remaining probability for the true value of a completely innocuous variable unrelated to the sensitive one. This second variable should have a similar range of possible values as the one under study to guarantee the privacy of the respondents within such a questioning design. After this first application, other techniques have been developed for the same purpose. The prerequisite for the possibility of applying such alternative questioning designs in statistical surveys is basic research and the elaboration of an applicable theory.
In the talk, a generalized framework for the estimation of the total of a sensitive variable is presented. It combines, as special cases, the most important techniques already published like the one from above as well as the direct questioning design under one theoretical umbrella. Furthermore, this unified approach at the same time encompasses all the other combinations not yet published. The execution of the statistical properties of this general framework for general probability sampling, being of the greatest importance for the practical application of the procedure, is based on unbiased “imputations” for the true values of the sensitive variables and the Horvitz-Thompson approach from statistical sampling theory.
Indirect techniques for surveying sensitive topics have been around since the 1960ies when Warner (1965) introduced the Randomized Response Technique. Since then, many different RRT variants as well as other approaches such as the Item Count Technique (ICT, Miller 1984) have been proposed. Along with the theoretical development of these techniques numerous methodological studies have been conducted to assess whether the indirect techniques actually generate more valid measurements than standard direct questioning. Results from these studies appear mixed, but a meta-analysis published in 2005 (Lensvelt-Mulder et al.) found that, on average, data generated by the RRT is indeed somewhat more valid than data from direct questioning. Since 2005, further RRT variants, such as the Crosswise Model RRT, have been introduced, applications of the Item Count Technique became popular, and a series of new and potentially more powerful validation studies have been conducted. We therefore carried out a new meta-analysis of RRT and ICT validation studies that covers publications in scientific journals indexed in major databases up to 2016. We present results on average effect sizes of particular techniques and variants and investigate whether results depend on study characteristics such as the employed validation design. In contrast to the 2005 meta-analysis we also investigate whether results are affected by publication bias.