All time references are in CEST
Methodological aspects of privacy measurement in surveys |
|
| Session Organisers | Mr Jošt Bartol (Faculty of Social Sciences, University of Ljubljana) Dr Andraž Petrovčič (Faculty of Social Sciences, University of Ljubljana) Dr Vasja Vehovar (Faculty of Social Sciences, University of Ljubljana) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Surveys are an indispensable tool for understanding pressing societal issues. One such contemporary problem relates to peoples’ privacy, as digital technologies continuously challenge established privacy norms. However, surveys can bring valid and actionable insights only when they are carried out in a methodologically sound manner. While this is true for surveys on privacy as much as for surveys on any other topic, there is little research on the development, validation, and deployment of surveys to measure privacy-related perceptions and behaviors. Advancing this line of research is important because perceptions of privacy are highly contextual and specific, with even subtle situational variations possibly leading to different evaluations of privacy and hence answers to corresponding questions. Moreover, personal and cultural factors can determine how people understand and act upon these situational cues. Research endeavors addressing these topics are likely to improve our understanding of participants’ response processes to privacy-related survey questions, enable deeper and more granular insights into privacy issues, and, in turn, provide a more robust basis for policymaking.
This session aims to facilitate discussion on how the conditions that govern the evaluation of privacy among individuals link to and influence their responses in surveys on privacy. To this end, submissions that deal with the following aspects are invited:
• Assessment of the impact of different survey modes on participants’ responses to survey questions about their privacy perceptions and behaviors.
• Examination of the role of contextual or situational cues, including potential order and framing effects, in responding to privacy-related survey questions.
• Investigation of cross-group (e.g., age, gender) and cross-cultural validity of privacy measures.
• Theoretical or empirical examination of respondents’ psychological processes that govern their responses to survey questions about privacy.
• Validation of survey scales for assessing privacy-related perceptions and behaviors in emerging digital contexts.
Keywords: cross-cultural validity, framing effects, measurement quality, order effects, privacy, response process, scale validation, survey mode, surveys
Mr Frieder Rodewald (University of Mannheim) - Presenting Author
Mr Florian Keusch (University of Mannheim)
Mrs Daria Szafran (University of Mannheim)
Mr Ruben Bach (University of Mannheim)
While surveys are an effective instrument to measure privacy-related attitudes, current research also uses surveys to measure participants' privacy behavior. Research shows that self-reporting behavior can suffer from recall and social desirability bias, thus potentially leading to inaccurate inference in subsequent analyses. This can be detrimental to privacy research, which typically uses privacy behavior as its dependent variable. This study investigates the extent of misreporting in survey questions about fine-grained Instagram privacy behavior by comparing it to objective measures collected via data donation. We also explore to what extent the accuracy of self-reports depends on the response format (labeled vs. numbered scale).
We will receive data donations from about 400 participants in a German probability-based online panel regarding over 20 distinct privacy behaviors on Instagram, including whether participants typically post content on which faces are visible, how they manage followers, and whether they share their personal contacts with Instagram. Participants first complete survey questions on these behaviors and we vary the response format within a split-sample experiment. Respondents are then instructed to download their Instagram usage data for the last three months and donate them to our research. We analyze correlation coefficients between behavioral self-reports and donated data to assess the accuracy of self-reports in general and for specific privacy behaviors.
We contribute in three key ways: First, we inform the field of questionnaire design by offering insights into how to accurately inquire about specific online privacy behaviors, which is particularly interesting for researchers who may not utilize data donation methods. Second, we examine the accuracy of self-reported data on individual Instagram privacy behavior, helping researchers assess the validity of surveying such behavior. Third, we illustrate the potential of data donation to gather detailed, fine-grained data on individual privacy behaviors.
Dr Frederic Gerdon (Mannheim Centre for European Social Research (MZES), University of Mannheim) - Presenting Author
Data about individuals can be used for a range of public benefit purposes, such as public health and infrastructure planning. Measuring public attitudes specifically towards such data use contexts is key for ethical evaluations from a privacy perspective. Moreover, given the internationalization of data markets and policies, it is becoming increasingly important to learn how attitudes and preferences regarding such policies vary context-specifically across countries. To be able to measure such variations, I combine Nissenbaum’s concept of Contextual Integrity with Masur et al.'s (2024) Comparative Privacy Research Framework to incorporate meso-level comparisons between social contexts into macro-level comparisons between countries.
I empirically applied this approach by conducting an international, longitudinal online survey experiment. For the meso-level, the survey experiment presented respondents with fictitious data use scenarios which varied in contexts and properties of the data use according to several Contextual Integrity parameters (data type, recipient, transmission principles). For the macro-level, the survey was run in three countries (Germany, Spain, and the United Kingdom) that vary in their degree of individualism, which could be relevant for attitudes towards using individual data for public benefit.
The findings reveal differences in effects of parameters across countries, particularly with respect to data recipients (such as a more negative effect of public agencies in Germany than in other countries). Across countries, the overall strongest effects are found for data type, showing a relatively higher acceptance for health data use. Beyond such concrete findings, this study demonstrates how exactly social contexts can be accounted for in macro-level comparative research on privacy attitudes. As the identified parameter-specific differences suggest, cross-country comparative research on privacy should consider context-specific comparisons additional to measurements of general privacy attitudes to reveal otherwise non-measured variations across countries.
Mr Jošt Bartol (Faculty of Social Sciences, University of Ljubljana) - Presenting Author
Dr Vasja Vehovar (Faculty of Social Sciences, University of Ljubljana)
Dr Andraž Petrovčič (Faculty of Social Sciences, University of Ljubljana)
Survey questionnaires are a valuable tool to study individuals’ perceptions and behaviors regarding privacy on the internet. However, little research has examined how questionnaire design influences individuals’ responses to privacy-related survey questions. In this study, we explore possible question-order effects when inquiring about online services use, self-disclosure, and privacy perceptions in relation to different online contexts and actors. Question-order effects occur when information presented in the preceding questions shapes participants’ understanding of subsequent questions. We focus on three online contexts (i.e., e-commerce, social network sites, and instant messaging) and two groups of actors (i.e., companies and acquaintances) and examine how the order in which the contexts and actors are presented to respondents influences their answers to the substantive questions. We recruited 465 respondents from an online access panel in Slovenia. To examine the question-order effects, we used a split-ballot design and then compared the mean scores between groups. Two split-ballots were used in the survey. In the first, we inquired about the use of and self-disclosure in the three online contexts but varied the order of the contexts. In the second, independent of the first, we inquired about privacy concerns, privacy control and trust in relation to companies and acquaintances, varying the order of actors. Comparison of the mean scores between groups revealed significant question-order effects. These were particularly pronounced for self-disclosure in different contexts. We also found question-order effects related to privacy concerns and privacy control in relation to acquaintances. Subsequent multi-group confirmatory factor analysis supported measurement invariance across the groups but suggested that not all associations between constructs were equal. Future research on these aspects is warranted as our findings indicate significant question-order effects when measuring internet-related privacy perceptions and behaviors.
Mr Yongchao Ma (University of Michigan) - Presenting Author
Professor Michael Traugott (University of Michigan)
This study investigates survey attitudes and privacy concerns in the context of cross-national surveys. Using data from a short 9-item survey attitude scale embedded in a survey on the US presidential election preferences and policy-related issues such as satisfaction with Medicare and Social Security systems, we replicated and extended prior research with the Understanding America Study (UAS), a probability-based internet panel representative of the United States. Our results confirmed that the same three-dimensional factor structure (enjoyment, value, and burden) identified in the original study also exists in the United States. Recognizing the sensitivity of the survey topics, a question on the invasion of privacy was included as an important indicator of survey burden. However, when assessing the measurement invariance of the scale across the three panels in Germany (GESIS), the Netherlands (LISS), and the United States (UAS), only configural invariance was supported. The original conclusion that the partial scalar measurement invariance held between Germany and the Netherlands was not supported by the results of the appropriate hypothesis testing in this study. Using the alignment method, we found approximate partial metric invariance across the three countries and approximate partial scalar invariance between Germany and the United States. Regarding the privacy item, approximate metric invariance held between Germany and the Netherlands, but scalar invariance did not. Specifically, for the same latent score on the burden factor, German respondents expressed significantly stronger views that opinion polls invade privacy compared to Dutch respondents (unequal intercepts), while US respondents expressed significantly weaker views compared to both German and Dutch respondents (unequal factor loadings). These results highlight the complexities of cross-national comparisons in surveys that measure privacy-related perceptions. This study offers insights for the design and interpretation of privacy-related survey questions in diverse contexts.
Dr Philipp Masur (Vrije Universiteit Amsterdam) - Presenting Author
Dr Kelly Quinn (University of Illinois at Chicago)
Dr Dmitry Epstein (The Hebrew University of Jerusalem)
Addressing the need for cross-cultural privacy research, this study revisits the Online Privacy Literacy Scale (OPLIS), adapting it from its original German context to the US. The adaptation of OPLIS for the US involved translating the scale and modifying items related to data protection law to align with the US legal system. The research uses representative survey data from Germany and the US to validate both 16-item and 12-item versions of OPLIS, demonstrating measurement invariance between the two countries. The scales assess four dimensions of online privacy literacy: knowledge of institutional practices, technical aspects, data protection law, and personal privacy strategies. The study confirms the reliability and validity of the scales, demonstrating that they can be a practical tool for cross-cultural application. Further, it demonstrates that privacy literacy predicts data protection behaviors in both countries, though with some differences. For example, while higher privacy literacy correlates with general privacy protection behaviors in both countries, specific behaviors, such as using TOR, showed country-specific relationships to privacy literacy. The findings underscore the importance of comparative research in understanding privacy literacy within diverse regulatory and cultural environments. The study also highlights the need to consider broader contextual factors when interpreting the relationship between privacy literacy and privacy behaviors.