All time references are in CEST
Interviewer-Assisted or Not? Strengths and Drawbacks of Mode Transitions in Times of Crisis |
|
| Session Organiser | Dr Melike Sarac (Hacettepe University Institute of Population Studies) |
| Time | Friday 21 July, 09:00 - 10:30 |
| Room | U6-01b |
There is a growing trend of switching modes of data collection from interviewer-assisted face-to-face surveys (e.g., PAPI and CAPI) to self-administered methods (e.g., web surveys and self-administered surveys). This is mostly because of the COVID-19 pandemic across the world in recent years as well as its subsequent challenges when collecting survey data (ESS, 2022). It appears to be that data collection procedures of survey organisations in times of crisis will have long-term effects, regardless of whether the pandemic or different kinds of crisis disappear completely.
As discussed in the literature on interviewer effects, interviewers utilize interviewing process by providing assistance to respondents although they may create bias on survey estimates. Socio-demographic characteristics, attitudes, and behaviors of interviewers might affect survey responses (West and Blom, 2017). On the other hand, matching characteristics between interviewers and respondents may improve survey data (Vercruyssen et al., 2017). From the total survey error perspective, interviewers are known as the principal contributors to measurement and non-response errors (West and Olson, 2010).
The methodological discussions about interviewer presence during the data collection are required while mode transition from interviewer-assisted to self-administered is among hot topics in such times. The submissions which aim to discuss the strengths and drawbacks of interviewer assistance during the data collection would be insightful for mode switching decisions in surveys. In this regard, the submissions are expected to suggest useful methodological strategies for future surveys to keep the overall quality at a high level.
Keywords: Mode transition, data collection, interviewer assistance, interviewer bias, crisis times,
Dr Wojciech Jablonski (Netherlands Interdisciplinary Demographic Institute) - Presenting Author
Professor Ignacio Pardo (University of the Republic)
Generation and Gender Survey (GGS) is a longitudinal, multidisciplinary, cross-national survey with a conceptual framework and an innovative research agenda designed to move from a descriptive approach to understanding the causal mechanisms of demographic and family change. The survey is managed by the Generation and Gender Programme (GGP) Central Coordination Hub at the Netherlands Interdisciplinary Demographic Institute in the Hague, the Netherlands (https://www.ggp-i.org).
Because of the COVID-19 pandemic, GGP introduced online data collection in most participating countries, eliminating or significantly reducing the face-to-face component. In Latin America, however, a region with a strong tradition of personal interviews, such switch into CAWI mode did not occur.
Within the presentation, we will focus on Uruguay where we decided to implement a mixed-mode approach. 85% of the sample (household sample N=8000, within household selection method: last-birthday) was conducted in CAPI mode, 15% – via CAWI. As the assignment was random, we can compare the response patterns between the CAPI and CAWI. The GGS in Uruguay was conducted by the Population Studies Programme (University of the Republic) and fieldwork finished in December 2022.
The presentation will briefly discuss the most interesting substantial outcomes and focus on methodological output, comparing the data from both CAWI/CAPI groups. We will also present qualitative insights based on interviewers' and field supervisors' observations.
Ms Laura Löwe (LIfBi) - Presenting Author
Mr Andre Pirralha (LIfBi)
Large-scale educational studies are an important resource to inform policymakers and the general public about the reach and effectiveness of diverse aspects of educational systems in several countries. Competence assessment in institutional settings (e.g. schools) has been an essential factor to collect valid measurements of cognitive abilities or motivations, for example. In order to conduct the assessment sessions, a significant number of test administrators (TAs) are necessary to supervise and coordinate test groups in the participating schools. The TAs undergo specific training and follow a strict protocol to ensure that competence assessment sessions are standardized and comparable so that student achievement data can be meaningfully collected. The TA characteristics can affect the quality of assessment scores and survey data. Differences in their behavior can originate interviewer effects, systematically impacting the validity and comparability of competence assessment tests. While there has been a recent effort to change competence assessment testing to computer-assisted modes of data collection, there is very little research aimed to uncover whether the training sessions and protocols are effectively delivering the goal of preventing TA effects in the first place.
In this paper, we explore the presence and magnitude of interviewer effects on paper-and-pencil competence assessments for mathematics abilities and survey questions in a German nationally representative longitudinal educational survey (National Educational Panel Study - NEPS). For this purpose, we will replicate the Lüdtke et al. (2007) paper, to date the only empirical investigation of TAs interviewer effects we are aware of. Multilevel analyses for cross-classified data are taken to effect to decompose the variance associated with differences between schools and the variance associated with TAs. The results are of use to improve competence assessment testing procedures, particularly by unveiling whether interviewer training and protocols should be improved and to assess the
Ms Anne Conolly (University College London / NatCen Social Research)
Mrs Soazig Clifton (University College London / NatCen Social Research) - Presenting Author
Miss Rebecca Light (NatCen Social Research)
Mrs Katharine Sadler (NatCen Social Research)
Professor Nigel Field (University College London)
Dr Clarissa Oeser (University College London)
Professor Pam Sonnenberg (University College London)
Professor Catherine Mercer (University College London)
Introduction
The fourth decennial Natsal survey (Natsal-4) of the British population, was under development when the COVID-19 pandemic began. Planned data collection involved in-home interviews (average length 1 hour) with interviewer-administered and self-completion questions, biosamples, and data linkage consent. Pilot fieldwork was postponed while alternative data collection methods were considered.
Methods
A risk/benefit review evaluated three data collection models:
1. Remote self-administration (eg push-to-web)
2. Remote interviewer-administration (eg push-to-telephone/video)
3. In-person interviewer-administration, with remote options (eg initial in-person contact with in-home or telephone/video interviews)
Each model was assessed against the survey design features (sample quality, time-series maintenance, boost samples, selection of one participant per household, interview length, self-completion questionnaire, biosampling, data linkage). Model 3 was developed and evaluated in two pilot studies (June-July 2021 and February-March 2022).
Results
Across the pilots, 30% (n=79/261) of interviews were conducted remotely (telephone (n=72); video (n=7)). Remote interviews included an online self-completion questionnaire, biosamples, and data linkage consent. Video interviewing was dropped following pilot 1 due to low uptake. Interviewers and participants were positive about the experience of administering / participating in remote interviews. Consent to biosamples, final biosample response rate (accounting for samples not received by the laboratory), and data linkage consent were all lower in remote interviews than in-home interviews (57% vs 64%; 34% vs 56%; and 61% vs 80%, respectively).
Discussion
Mainstage Natsal-4 fieldwork retained a remote (telephone) option, but emphasised in-home interviewing as the preferred mode. Natsal has demonstrated it is feasible to adapt a complex interviewer administered survey to enable remote data collection, however substantial modifications were needed and response to biosampling and data linkage were lower in remote interviews.
Mr Florian Heinritz (Leibniz Institute for Educational Trajectories & Universität Hamburg) - Presenting Author
In every crises, the political debate and also surveys often focus on groups that previously received less attention. For example, in the second half of the last century, the focus turned on refugees. Unlike the Ukrainian refugees, the group of refugees in the last decade was a rather heterogeneous group.
When discussing face-to-face interviews with interviewers versus self-administered interviews for this group of refugees, this heterogeneity becomes particularly apparent in two aspects: On the one hand, the refugees came from different countries and therefore spoke different languages; on the other hand, the literacy rate in most countries of origin was significantly lower than in Europe.
Both aspects have conflicting implications for the considerations of whether or not to use interviewers. Since most newly arrived refugees rarely have sufficient knowledge of the host country’s language, it is necessary to offer the interviews in (at least) the most common native languages of the refugees, so that no group will be excluded from taking part in the survey. This can result – in addition to the anticipated and well-studied effects of interviewers – in challenges to recruit interviewers in the required languages for face-to-face interviews. At the same time, it is important to ensure that illiterate respondents are not excluded in self-administered interviews with refugees.
Using data from the ReGES study conducted in Germany, I will first explain why interviewers make sense for this group despite the interviewer effects and the high effort required for multi-language interviews. I will also discuss alternative strategies for interviewing illiterate people in self-administered interviews. Since the ReGES study was conducted in several waves and different modes, I will then compare those survey waves with and without interviewers to show pros and cons of both modes.