Video interviewing for social surveys: before, during and beyond the pandemic (session 2) |
|
| Session Organisers |
Mr Tim Hanson (European Social Survey ERIC (City, University of London)) Professor Frederick G. Conrad (University of Michigan, USA) |
| Time | Friday 9 July, 16:45 - 18:00 |
The COVID-19 pandemic has posed numerous challenges for surveys, especially those that use face-to-face in-person interviewing. As a result, many surveys have adapted to use alternative or complementary methodologies. This includes video interviewing, where interviewers can use video software platforms (e.g. Zoom, Microsoft Teams) to interview respondents remotely while retaining face-to-face contact and the ability to screen-share survey materials.
This session will present international experiences and evidence from video interviewing, including studies conducted before and during the pandemic. Topics to be covered include: different approaches used for video interviewing; key design decisions and trade-offs; levels of take-up of video interviews across different surveys; respondent and interviewer experience; data quality indicators; lessons learnt to date; and expectations and recommendations for the future of video interviewing (including beyond the pandemic).
Keywords: video interviewing, remote interviewing, COVID-19 pandemic, user experience, recruitment, mode comparison, data quality
Ms Kirsty Cole (NatCen Social Research) - Presenting Author
Ms Samantha Spencer (NatCen Social Research)
Ms Lucy Haselden (Centre for Longitudinal Studies, UCL)
Mr Matt Brown (Centre for Longitudinal Studies, UCL)
Dr Erica Wong (Centre for Longitudinal Studies, UCL)
Mr Martin Wood (NatCen Social Research)
Ms Angela Stockman (NatCen Social Research)
This paper will explore the findings of a small pilot of video call interviews conducted on the Age 50 Survey of the 1970 British Cohort Study (BCS70) and its wider learnings and implications for video call interviewing as a future methodology.
BCS70, which follows the lives of around 17,000 people, was preparing for main-stage fieldwork when the pandemic restrictions put a stop to face-to-face data collection. This led to discussions on the best way to continue the research in a way that would be Covid secure, encourage respondent participation and collect high quality survey data. BCS70 is a long questionnaire with complex modules and sensitive content which does not readily lend itself to be adapted easily to telephone or web modes without limiting much of the data collected. Computer-Assisted Video Interviewing (CAVI) was felt to be a viable option where an in-home interview is not possible. It maintains a human “face-to-face" interaction and requires minimal changes to procedures and content developed for in-person interviewing (including showcards and cognitive assessments).
In October 2020, a small-scale feasibility test of video interviewing was completed by NatCen and Kantar interviewers using MS teams. We selected a sample of 60 cohort members to take part in a CAVI interview with no alternative methodology offered. The sample was intentionally skewed towards those more engaged in the study and half of the respondents had indicated a willingness to do a CAVI interview in a separate online survey. The pilot findings were positive with forty-four cohort members taking part.
This paper will discuss some of the adaptions made to the face-to-face survey, to make it suitable for CAVI. It will then examine the main findings from the pilot and their implications for other surveys, particularly: respondents’ willingness to take part in this approach, how well the technology and sharing of materials (such as showcards) worked overall and on different devices, and the length of interview that can be offered.
We will discuss some of the learnings from the pilot such as how to improve the participant experience; best practice for video call interviewing; and training needed for face-to-face interviewers to adapt to this new approach. We will also consider what we would like to learn next about CAVI as a methodology and what our next steps will be. We are also aiming to present some of the results of a second pilot of CAVI interviewing on the Age 60 cohort of the National Child Development Study, which plans to test CAVI in combination with face-to-face interviews. This test is planned to take place in spring 2021 but has not yet been confirmed due to COVID 19 restrictions.
To conclude we will look at the advantages and disadvantages of video call interviewing and how useful it is likely to be to other studies and in what circumstances.
Ms Kallan Larsen (University of Michigan) - Presenting Author
Mr Andrew Hupp (University of Michigan)
Professor Frederick G. Conrad (University of Michigan)
Ms Ai Rene Ong (University of Michigan)
Professor Michael F. Schober (The New School)
Professor Brady West (University of Michigan)
Ms Tianheao Wang (University of Michigan)
The COVID-19 pandemic has prompted survey researchers to turn to video technology as an alternative to in-person data collection. Despite a widespread increase in familiarity with video technology, the novelty of video as a survey mode leaves many questions unanswered regarding best recruitment practices and patterns of participation. Live video interviews, while much like phone interviews, require a recruitment process more similar to in-person interviewing: respondents must schedule a time at which to meet with interviewers, and “cold calling” is not a feature of most video platforms. Self-administered recorded-video, where questions are asked via pre-recorded videos of interviewers embedded in a web survey (rather than written text), requires respondents to have audio capabilities and can restrict the speed at which an interview is completed relative to a traditional web survey (because of the time it takes the videos to play). From August 2019 to March 2020, we conducted a mode comparison experiment using both an address-based sample and two online non-probability panels as recruitment sources (n = 1083 completed cases in total). We compare recruitment and participation patterns (including breakoffs) for each sample source across three modes: live video, recorded-video, and traditional (text-only) web. We examine the apparent impacts of device type and respondent age on completion rates. We did not find ABS to be an effective recruitment method for live video interviews. Recruitment was most difficult for live video in online panels as well, and breakoffs were most common in the recorded video mode and particularly infrequent in live video. As with any mode, there are advantages and disadvantages to video modes, and we will briefly discuss these tradeoffs as they relate to data quality.
Mr Tom Nickson (Kantar) - Presenting Author
Ms Alice McGee (Kantar)
Ms Becky Hamlyn (Kantar)
Since early 2020, Covid-19 has imposed restrictions and mandated the need for change and adaptation across many areas of research methodology. For qualitative research methods such as cognitive and usability testing, depth interviews and focus groups, traditional approaches to in-person interviewing were suddenly no longer possible. Subsequently, we urgently needed to find more flexible and innovative approaches.
Traditionally, qualitative research has involved in-person interaction, with one or a group of participants, to facilitate rapport and a deeper understanding of the thoughts of those being interviewed. The pre-testing of survey questionnaires has also traditionally been carried out qualitatively, including both cognitive interviews (testing understanding of survey questions) and usability testing (exploring interaction with a self-completion questionnaire).
The potential benefits of remote interviewing were recognised before Covid-19, due in part to the increased availability and functionality of online video conferencing platforms. Kantar had already begun exploring the feasibility of these methods but Covid-19 meant we needed to develop a workable solution much more quickly, video interviewing providing the only tangible option available.
This paper will discuss the practicalities and logistics of conducting different types of qualitative and pre-testing interviews via video before setting out the key challenges and benefits experienced and sharing ways we overcame obstacles.
Challenges included: logistical issues such as ensuring appropriate consent and reducing the size of focus groups for ease of management; technical issues (e.g. internet access and connection, confidence in using the technology, and online features such as screensharing for usability testing); and communication issues (such as greater difficulty in establishing rapport and keeping respondents engaged and motivated).
Benefits included: sample diversity (e.g. ability to target a wider geographical spread and include participants who would be unable to travel to interviews or groups in-person); reduced cost and increased flexibility of both recruitment and fieldwork; online features that allow focus groups to be facilitated in more tailored and bespoke ways; greater potential for interviews to be observed by clients or other researchers; and the ability to video-record interviews to allow detailed analysis. The requirement to find an alternative to in-person qualitative interviewing meant that we were given the push we needed to become more flexible in our approach to qualitative interviews.
While video interviewing has provided a useful and essential alternative to in-person interviewing over the last year, we would not recommend that this fully takes the place of in-person interviewing in the longer term. We see video interviews as a complementary option to in-person interviewing, providing access to hard to reach or geographically dispersed groups or providing opportunities to work in a more agile way where there may be restrictions in terms of budget or time.
Based on our experiences of remote interviewing across several studies, this paper will conclude by putting forward a full set of recommendations for conducting qualitative and pre-testing interviews via video and will outline our thoughts going forward for further innovation in this space.
Ms Shlomit Okon (The New School) - Presenting Author
Professor Michael F. Schober (The New School)
Professor Frederick G. Conrad (University of Michigan)
Ms Kallan Larsen (University of Michigan)
Ms Ai Rene Ong (University of Michigan)
Mr Andrew Hupp (University of Michigan)
Given the necessity of considering alternatives to in-person data collection in the COVID moment, when and how people will be willing to participate in standardized surveys that inform policy has become critically important. The study presented here tests hypotheses about four potential predictors of respondents’ willingness to participate in surveys via the live video that has become ubiquitous for many members of the public in the COVID era--at least for those who have access and sufficient connectivity. Three potential predictors (selecting factors that have predicted technology acceptance and behavioral intention in other communication modes) are participants’ perceptions about using live video for communication in contexts other than surveys, in particular (1) how easy they find live video to use, (2) how useful they find live video, and (3) how much they enjoy using live video. A fourth hypothesized (survey-specific) predictor is the extent to which participants report that they (4) would be uncomfortable answering a particular sensitive question in live video relative to other survey modes. In the study, online participants rate their willingness to take part in a hypothetical live video survey that might ask about personal information in comparison with their willingness to take part in four other survey modes that differ from live video in theoretically and practically important ways: in-person, phone, a traditional (text-only) web survey, and a “prerecorded-video” web survey in which respondents play embedded video of interviewers reading questions. The modes vary in whether they are interviewer- or self-administered, in the social presence of the interviewer (which in video could be either intrusive or rapport-building), and in the interactive nature of the survey dialogue. Findings will be informative about the feasibility of live video interviewing for large-scale data collection moving forward.