Various innovations and insights in times of COVID-19 |
|
| Session Organiser | Mr Boris Sokolov (National Research University Higher School of Economics) |
| Time | Friday 23 July, 13:15 - 14:45 |
This session focuses on the discussion of various innovations in, and adjustments to, surveys provoked by the ongoing COVID-19 pandemic. The specific topics covered by the presentations selected for this panel, are the following:
(1) The pandemic's effects on the methodology of household surveys, in terms of msot noticeable changes in prevalent survey modes, respondent recruitment methods, ethical standards and other, important techinical details.
(2) How the standard approaches to question evaluation can be modified in the face of the pandemic (exempilfied by the experience of the US National Center for Health Statistics).
(3) A German-language adaption and validation of the Tolerance for Ambiguity Scale (TAS)
(4) The pandemic's impact on research agendas, publication activities, and science communication strategies in various scientific disciplines.
Keywords: COVID-19 pandemic, innovations in surveys, question evaluation, cognitive interviewing, web probing, measurement error, scale adaptation, tolerance to ambiguity, science communication
Dr Tugba Adali (Hacettepe University Institute of Population Studies) - Presenting Author
Mr Yaser Koyuncu (Hacettepe University Institute of Population Studies)
Professor Ahmet Sinan Türkyılmaz (Hacettepe University Institute of Population Studies)
In this study, we aim to review the effects of the pandemic on household surveys, on which the survey research community had actively focused in 2020. Our desktop review included the special issue of SRM on this topic, webinars by professional organizations and universities (WAPOR, AAPOR, etc.), websites of international organizations (UN Statistics Division, EUROSTAT, etc.), and websites of some international survey series (DHS, MICS, Gallup, etc.).
Our review showed us that the pandemic's effect depends on the country (developing countries vs. developed countries). Common remedies included postponing surveys, mode switches, introducing rapid surveys, and using other data sources to provide data. We observed that timing mattered in terms of at which stage a particular survey was when the pandemic was announced. Longitudinal surveys had better chances of being realized with contact information being available from previous waves, compared to cross-sectional ones. By mode, surveys relying on face-to-face mode were affected the most, followed by those relying on centralized CATI, with limited impact on mail surveys. Mode switches were also applied to mixed mode-surveys, with for instance CASI components to CAWI or postal ones; indicating the versatility of such designs. We saw some innovation in respondent recruitment, such as those relying on social media users. Another observation was that new ethical concerns come to the fore; consent procedures had to be rethought for surveys which switched modes, surveys that kept FTF components had to ensure safety measures, new issues were discussed for contact tracing apps, and some topics were deemed off the table unless FTF visits were done.
One of our take-aways was the importance of having setups that do not rely on FTF (such as longitudinal CATI surveys or online panels) for each country. Another was that other data sources, such as administrative data or other sources of big data, such as Google Mobility reports, could provide complementary information to survey statistics for statistical offices and researchers. We also noted that the pandemic had potential error implications on surveys related to coverage, nonresponse, and measurement, which data users from this era will need to keep in mind.
Dr Kristen Cibelli Hibben (National Center for Health Statistics) - Presenting Author
Dr Paul Scanlon (National Center for Health Statistics)
At the beginning of the COVID-19 pandemic, the United States’ National Center for Health Statistics (NCHS) was able to modify the purpose of its Research and Development Survey (RANDS) to quickly collect and produce statistics on coronavirus-related topics in timely manner, allowing NCHS to release data on the pandemic at a time when some of its traditional household surveys were unable to do so. The concepts surrounding the COVID-19 pandemic and the public’s response to the outbreak were new; in the normal course of business, NCHS would conduct an extensive and systematic cognitive evaluation of these new items before they were fielded. However, with in-person cognitive interviews impossible and virtual cognitive interviews not yet available to NCHS, the RANDS questionnaire was developed without this important pretesting. Therefore, in order to understand how these new, coronavirus-related survey items functioned—and to determine what constructs they captured—RANDS included a series of open- and close-ended web probes and experimental designs, which provided data for post hoc evaluations.
This presentation will explain how NCHS was able to modify its question evaluation approach in the face of the pandemic. Using a series of examples including items about telemedicine access, quarantine behavior, and reference period comprehension, we show how we modified our typical evaluation methodology combining cognitive interviews and web probes to one using primarily web probing. Furthermore, we will demonstrate how we incorporated computer-assisted approaches including natural language processing into our analytic processes in an effort to manage the large amount of qualitative data produced by these data collection steps, and will explain the steps we took to validate these computer-assisted analysis approaches.
Dr Kerry Levin (Westat) - Presenting Author
Dr Jocelyn Newsome (Westat)
Dr Jill De Matteis (Westat)
Dr Maeve Gearing (Westat)
Dr Jon Wivagg (Westat)
Once the COVID-19 pandemic hit the United States (U.S.), social science researchers began scrambling to field studies quickly. The goal was to gather time-sensitive information during an unprecedented time. Worldwide, about 1,800 social science articles about COVID-19 were published in the first half of 2020 (Aristovnik, Ravšelj, & Umek, 2020). Given the urgency, most U.S. studies opted to utilize panels and non-probability panels, resulting in findings not generalizable to the U.S. population.
Westat collaborated with Stanford University’s School of Medicine to field one of the first nationally representative surveys on COVID-19. The development of the survey, which focused on the social, health, and economic impact of COVID-19, began in April and the survey was fielded in May 2020. Survey topics included: beliefs about the risk of contracting COVID-19 and probable health outcomes; how people’s behaviors had changed since the pandemic began; the influence of stay-at-home orders on those behaviors; and the economic impact of the pandemic. The study utilized address-based sampling and a single mode of data collection; the web survey collected both quantitative and qualitative data.
This paper will discuss the process of deciding upon the methodology of the study, the experience of fielding the study, and selected findings. For example, we will discuss findings surrounding the financial impact of the pandemic, concerns voiced by respondents, and coping strategies.
References
Aristovnik, A., Ravšelj, D., & Umek, L. (2020). A bibliometric analysis of COVID-19 across science and social science research landscape. Sustainability, 12(21), 9132.
Ms Almuth Lietz (research associate) - Presenting Author
Diversity in lifestyles and opinions or in terms of social groups and religions shape life in Germany. Whether diversity poses a challenge to social cohesion or instead improves society’s potential for innovation and creativity depends in particular on how a society values diversity. A central variable related to this is the tolerance for ambiguity. Originally defined as a stable personality trait that is independent of situation and context, it is now more often assumed that tolerance for ambiguity can change over time and can even be learned. Furthermore, it is partly understood as a more context-specific construct. Although it is an important variable with regard to the acceptance of diversity and thus appears essential for the functioning of a society, it is rarely asked in population surveys in Germany. Moreover, hardly any validated measurement instruments exist in the German language that are also suitable for population surveys, which are often restricted by time and monetary constraints. Therefore, an acceptable validated English-language measurement instrument was translated into German and tested. Building on this, a short scale for measuring tolerance for ambiguity is proposed and validated. The development is based on the English Tolerance for Ambiguity Scale (TAS), which is surveyed with 12 items covering four subdimensions: valuing diverse others, change, challenging perspectives, and unfamiliarity. The items were translated into German using the TRAPD approach and tested within a quantitative pretest (quota sample). As a development sample, the instrument is being tested as part of the cooperation project "Social cohesion in times of crisis. The Corona Pandemic and Anti-Asian Racism in Germany". The tracking study is composed of three waves that are used to shorten the instrument on empirical criteria and expert discussions. In order to evaluate the psychometric quality of the translated TAS and the short scale, reliability (McDonald’s omega, test-retest stability) and validity (factorial, convergent and divergent validity) are considered. First results (N=2,643) show an internal consistency of ω=.71 for the overall 12-item TAS.
Mr Gregor Fabian (DZHW) - Presenting Author
Mr Jens Ambrasat (DZHW)
Science communication is becoming increasingly important, not only for the media and politicians, but also for scientists themselves, who are investing more and more time to communicating their research results. The Covid-19 pandemic has further intensified the discussion about the role of science communication and the relationship between science and politics. There has been and still is much debate in the media and in society about science, its role in policy advice, the public appearances of individual scientists, but also partly about the functionality and the rationality of established scientific practices.
What is largely missing is the perspective of the scientists itself, their own involvement in science communication and their perceptions and assessments of the science policy-relationship. Likewise, it is interesting in how far research practices and science communication activities changed due to the pandemic. Answers are provided by the data from a very recent survey conducted by the DZHW among researchers on the topic of science communication. At the end of 2020, 5,688 scientists from German universities and non-university research institutions (AFE) were surveyed. The survey asked about the practices and attitudes of scientists in the field of science communication, their perceptions of the relationship between science and politics, and the role of the media.
The data of the Research Survey provide for the different disciplines insights into how strongly the work of scientists is influenced by the Covid 19 pandemic, to what extent the pandemic has an impact on the researchers’ research agendas, publication activities, and their science communication. We also asked the researchers about their attitudes towards the relationship between science and policy and the role of scientific experts in the public and media.