Video interviewing for social surveys: before, during and beyond the pandemic (session 1) |
|
| Session Organisers | Mr Tim Hanson (European Social Survey ERIC (City, University of London)) Professor Frederick G. Conrad (University of Michigan, USA) |
| Time | Friday 2 July, 16:45 - 18:00 |
The COVID-19 pandemic has posed numerous challenges for surveys, especially those that use face-to-face in-person interviewing. As a result, many surveys have adapted to use alternative or complementary methodologies. This includes video interviewing, where interviewers can use video software platforms (e.g. Zoom, Microsoft Teams) to interview respondents remotely while retaining face-to-face contact and the ability to screen-share survey materials.
This session will present international experiences and evidence from video interviewing, including studies conducted before and during the pandemic. Topics to be covered include: different approaches used for video interviewing; key design decisions and trade-offs; levels of take-up of video interviews across different surveys; respondent and interviewer experience; data quality indicators; lessons learnt to date; and expectations and recommendations for the future of video interviewing (including beyond the pandemic).
Keywords: video interviewing, remote interviewing, COVID-19 pandemic, user experience, recruitment, mode comparison, data quality
Professor Frederick J. Conrad (University of Michigan) - Presenting Author
Professor Michael F. Schober (The New School)
Mr Andrew Hupp (University of Michigan)
Professor Brady West (University of Michigan)
Ms Kallan Larsen (University of Michigan)
Ms Ai Rene Ong (University of Michigan)
Ms Tianheao Wang (University of Michigan)
Since video capability has become standard on computers and smartphones, video communication has become ubiquitous--at least for those with access to the right equipment and adequate connectivity. To what extent might video technologies be useful for survey measurement? For studies that require real time, face-to-face interaction between respondents and interviewers because, for example, difficult response tasks require interviewers to help respondents or to keep respondents engaged and attentive over a long interview, live video interviewing might be a viable alternative to in-person interviews. In this study we compare respondents’ (online panelists’) performance and subjective experience in live video-mediated interviews to two self-administered modes, one in which prerecorded videos of the same interviewers who conducted the live video interviews asking the same survey questions are embedded, and a conventional (textual) web survey. The findings demonstrate significant advantages and disadvantages for data quality for both video modes relative to the conventional web survey. Respondents assigned to live video interviews straightlined less (producing higher quality data) than respondents assigned to web surveys, but they provided rounded numerical responses more, disclosed less sensitive information, and left more questions unanswered, especially when the questions concerned sensitive topics (lower data quality). Live video respondents also reported significantly higher satisfaction with their experience completing the survey than for either of the self-administered modes. This pattern of findings is consistent with a view that live video is much like in-person interviewing, for better or worse: an interviewer’s “presence” can keep respondents motivated and conscientious in some situations, but may also lead respondents to answer in ways that reflect the time pressure of in-person interaction and to shade the truth, presenting themselves more positively because the interviewer’s presence and ability to judge the respondents feels real even if mediated. At least as we implemented them and with these participants, both video modes seem viable for production survey data collection, which may be of particular value when in-person interviews are not feasible.
Mr Tim Hanson (European Social Survey ERIC (City, University of London)) - Presenting Author
Mr Niccolo Ghirelli (European Social Survey ERIC (City, University of London))
The COVID-19 pandemic has presented many challenges for surveys that usually rely on in-person data collection. This includes the European Social Survey (ESS), which has been conducted biennially across over 30 countries since 2002, with all countries delivering in-person samples at each round based on a central specification.
Following consideration of alternative approaches, it was decided to continue to plan for in-person data collection at Round 10 of ESS (originally due to be carried out between September 2020 and January 2021), but to expand the fieldwork period to the end of 2021. The decision to retain an in-person approach was based on the importance of a mode-consistent time series (especially in the context of understanding the impact of the pandemic on attitudes and behaviour), and a view that no alternative methodology could be delivered consistently by all 30+ countries to an equivalent standard.
It was however acknowledged that the pandemic may impact on the participation of certain groups in an in-person interview, even after a return to relative normality. We have therefore adapted the ESS specification to allow video interviews to be conducted in cases where target respondents are unable or unwilling to be interviewed in-person. We expect video interviews to be implemented by 10-15 countries at ESS Round 10, as a complementary method to the usual in-person approach.
This paper will share key features of our approach to video interviewing, initial findings on the prevalence and success of video interviews in countries, and future learning based on our experience to date.
We will also briefly describe other innovations on ESS, including development of self-completion modes, linked both to the COVID-19 pandemic and longer-term concerns about the continued feasibility of in-person fieldwork in some countries.
Dr Gudbjorg Andrea Jonsdottir (University of Iceland) - Presenting Author
Dr Sigrun Olafsdottir (University of Iceland)
Mr Helgi Gudmundsson (University of Iceland)
Mr Arni Bragi Hjaltason (University of Iceland)
COVID-19 has had a tremendous impact on various aspects of society, not least the way we communicate. We have seen an accelerated implementation and use of online communication tools in areas such as work, education, healthcare, entertainment and leisure, and online commerce. Survey researchers that hitherto have relied on face-to-face interviewing, often regarded as the gold standard of data collection, have been forced to rely on other data collection modes such as web-surveys or video interviewing.
This paper discusses the data collection in the ISSP 2019 Social Inequality module conducted in Iceland from July 2020 to January 2021. A random sample of 3148 individuals was drawn from the National Population Register. An advance letter introducing the choice between participating through a face-to-face interview or a video call via a computer was sent by mail. The letter was followed up by a telephone call to schedule an interview of respondents’ choice. After accounting for 145 ineligible individuals in the sample, a net response rate (AAPOR RR1) was around 40% with 1194 completed interviews. Just under 20% of respondents or 222 individuals chose a video call. The Whereby platform was selected for the video calls due to its user friendliness. Contrary to alternative solutions, Whereby does not require the respondent to login or install any software. Those who chose to be interviewed via a video call were overall younger (m = 42.4) than those who answered by other modes of data-collection (m = 52.3). Furthermore, a video interview was more popular among respondents with a tertiary degree (24.5%) than primary or secondary education (14%).
The paper explores what distinguishes respondents choosing different interviewing modes and how the option of a video call changes the representativeness of the survey. Furthermore, the analysis tracks changes in the number of COVID-19 cases in Iceland, incidence rate and restrictions on gatherings throughout the data collection period and tests whether these measures are reflected in the refusal rate in the survey or the number of respondents opting for a video call.
Dr Lauren Guggenheim (University of Michigan) - Presenting Author
Dr Natalya Maisel (Stanford)
Mr David Howell (University of Michigan)
Ms Michelle Amsbary (Westat)
Professor Ted Brader (University of Michigan)
Dr Mathew DeBell (Stanford)
Ms Cindy Good (Westat)
Professor D. Sunshine Hillygus (Duke University)
Since 1948, the presidential time series conducted by the American National Election Studies (ANES) has included a face-to-face component. In 2020, however, challenges related to the COVID-19 pandemic led to the replacement of in-person interviewing with a contactless, mixed-mode design that included surveys self-administered by web and surveys administered by interviewers via live video or phone. Each protocol used the same nationally representative sample, selected using probability methods from an address-based frame. Sampled households were mailed hard-copy invitations and asked to complete an online screener, after which a respondent was randomly selected from the household and subsequently assigned to participate by Zoom video conferencing or by a self-administered web questionnaire. Telephone interviews were presented as an option later in some protocols to allow the participation of respondents uncomfortable with, or unable to access, web-based or video interviewing. Several interventions were implemented across all modes to increase response rates including escalating incentives, and there were also interventions specific to video such as the use of a specially trained refusal conversion team.
Our focus here is specifically on the impact of the use of video on recruitment and participation, including how potential respondents and interviewers reacted to and perceived video interviews. To do so, we employ several types of data collected over the duration of the survey including interviewer observations, differential participation by demographics and political attributes, help desk logs, participant emails, and a survey of interviewers. We supplement this by comparison to data previously collected by ANES and against groups that participated by a mode other than video in the current study. We additionally discuss design decisions such as whether to schedule appointments or have interviewers standing by, and their implications for participation. We further reflect on how the pandemic may have influenced participation across the different modes.