Detecting, Explaining and Managing Interviewer Effects in Surveys 2 |
|
| Session Organisers | Dr Daniela Ackermann-Piek (GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany) Mr Brad Edwards (Westat) Dr Jette Schröder (GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany) |
| Time | Thursday 18th July, 09:00 - 10:30 |
| Room | D16 |
How much influence do interviewers have on different aspects of the survey process and how can we better reduce their negative impact on the data quality as well as enhance their positive impact?
Although interviewer effects have been studied over several generations, still, interviewer effects are of high interest on interviewer-administered surveys. Interviewers are involved in nearly all aspects of the data collection process, including the production of sampling frames, acquisition of contact and cooperation with sample units, administration of the survey instrument, and editing and transition of data. Thus, interviewers can cause errors and prevent errors in nearly all aspects of a survey.
However, the detection of interviewer effects is only a first step. Thus, it is of interest to understand why interviewer effects occur. Although there are various studies explaining interviewer effects using multiple sources of data (e.g., paradata, interviewer characteristics, response times, etc.), the results are inconclusive. In addition, it is essential to prevent negative interviewer effects before they occur to ensure that interviewer-administered surveys can produce high-quality data. There are multiple ways to intervene: interviewer training, monitoring during fieldwork, adaptive fieldwork design or switching the survey mode, etc. However, still, relatively little is known about how all these different methods can effectively reduce interviewer error because there is a lack of experimental studies.
We invite researchers to submit papers dealing with aspects of detecting, explaining and preventing interviewer effects in surveys. We are especially interested in quasi-experimental studies on the detection, explanation, and prevention of interviewer error in surveys, and on the development or encouragement of interviewer ability to repair or avert errors. We welcome researchers and practitioners from all disciplines across academic, governmental, private and voluntary sectors to contribute to our session.
Keywords: Interviewer effects, Interviewer training, Interviewer characteristics, Paradata, Total Survey Error
Mrs P. Linh Nguyen (University of Essex - University of Mannheim) - Presenting Author
In Sub-Saharan Africa, where only one in five people uses the Internet and connectivity issues restrict the possibility for phone surveys in rural areas, interviewer-administered face-to-face (F2F) surveys are and will remain the principal data collection tool in the foreseeable future. Yet questions remain as to what extent previous findings on interviewer-administered surveys from Western countries may apply to a different cultural and geographical context. In this light, the objective of this study is to investigate the influence of certain observable interviewer characteristics (such as gender, age) and non-observable characteristics (such as education, attitudes) on interviewer variance on a subset of survey questions – both factual and attitudinal ones. Due to the different societal structure in Zambia, respondents may potentially interpret and respond to the cues given by the interviewer in a different way. The analysis draws on data from a face-to-face survey on standards of living, economic situation and financial behaviour in rural or semi-urban areas of Zambia. The survey was administered in 2016 with more than 2,000 members of selected collective savings groups who are beneficiaries of a development programme. For each savings group, a team of five interviewers were randomly assigned to randomly selected respondents following a partially interpenetrated assignment. Previous literature stresses the importance to differentiate interviewer variance introduced during the recruitment and nonresponse stage from that related to measurement error. In this study, not the interviewer but supervisors with support by savings group leaders select and ensure participation of respondents, as well as carry out the assignment of interviewers. This particular design allows to focus on interviewer effects on measurement primarily as the interviewer was not involved in recruitment. This study on interviewer effects presents both the interviewer variance analysis on the selected questions, as well as the results of multi-level models.
Ms Alexandra Cronberg (Kantar Public) - Presenting Author
Dr Charles Lau (RTI International)
Ms Pallavi Dhall (Kantar Public)
This study seeks to assess the effect of gender of interviewer on responses to gender-related questions in a Computer Assisted Telephone Interview (CATI) survey in India. Based on previous studies (Lipps & Lutz, 2016; Huddy et al. 1997; Groves et al., 1992), we hypothesised that female interviewers would elicit more feminist responses and that women would respond in a more feminist manner than men.
Kantar Public interviewed 1800 adults (1643 men and 157 women) in Haryna and Uttar Pradesh in 2017 using random digit dialling. Interviewers were randomly allocated to respondents. The questionnaire included four question on gender-related issues: 1) Should men share housework with women? 2) How often are women treated unequally by employers? 3) Should women’s share of inheritance equal that of men’s share? 4) Are men justified in hitting their wives?
Gender of interviewer had a statistically significant effect (p<0.05) on the questions relating to unequal treatment by employers, inheritance, and gender-based violence, but not housework. More respondents agreed that women should get an equal share of inheritance (86% male interviewer vs 91% female interviewer, Chi-square=0.001) and disagreed that men are justified in hitting their wives (16% male interviewer vs 12% female interviewer, Chi-square=0.030) when asked by a female interviewer. In contrast, a different effect was observed for the factual question (treatment by employer). Compared to responses from the matched-sex interviews (33% of both male and female respondents reported 'often' unequal treatment), female respondents overplayed unequal treatment to male interviewers, whereas male respondents underplayed it to female interviewers (45% and 23% reported ‘often’, respectively, Chi-square=0.000).
We conclude the expected gender-of-interviewer effect is partially observed in these states. For factual questions the ‘feminist effect’ may be outweighed by a desire to over- or underplay inequalities among women and men, respectively.
Dr Gundula Zoch (Leibniz Institute for Educational Trajectories) - Presenting Author
While an increasing number of studies investigates short-term changes in individual-level gender ideologies, most of these studies seem to have largely overlooked whether and to which extend changes in respondents’ self-reported gender ideologies are driven by interviewers’ characteristics. Using panel data, this study seeks to contribute by, firstly, investigating whether interviewer gender affects respondents’ self-reported gender ideologies and secondly, by assessing effect heterogeneity with respect to respondents’ and interviewers’ characteristics as well as interview mode. Based on social desirability theories, respondents are presumed to report more egalitarian views to female interviewers. According to the conditional attribution model, effects are expected to be more pronounced among some respondents, such as males, younger and highly educated respondents and among those, being interviewed face-to-face, and by younger and/or highly educated female interviewers. The analysis uses data from the National Educational Panel Study (NEPS) (adult cohort) and applies fixed effects models. NEPS respondents are interviewed by different interviewers and with different interview mode in alternate years, thus, providing sufficient variation in interviewer characteristics. Preliminary findings show that respondents report significantly more egalitarian views to female interviewers, with more pronounced effects among male respondents. Among those, interviewer gender effects seemed to be more likely in computer-assisted personal interviews compared to telephone interviews and for younger respondents. For both male and female respondents, the effect was mostly statistically significant only for the more egalitarian slanted attitudes compared to items that use a comparatively conservative language. Overall, the first results point towards heterogeneous interviewer gender effects and, thus, a more pronounced social desirability bias among certain subgroups, providing support to the conditional attribution model in the field of gender ideologies. Given this preliminary evidence, further analysis will conduct more in-depth analyses on the interaction between interviewer gender and survey mode as well as (further) respondents’ and interviewers’ characteristics.
Professor Geert Loosveldt (KU Leuven) - Presenting Author
Dr Caroline Vandenplas (KU Leuven)
A frequently used procedure to evaluate interviewer effects is the calculation of the intra interviewer correlation coefficient (IIC) based on a two-level random intercept model which take into account the nesting of the respondents within the interviewers. The specification of such a basic multi-level model does not really allow to investigate the relationship between certain respondent characteristics and the extent to which these characteristics influence interviewer effects. However it is reasonable to assume that some respondents are more sensitive to interviewer effects and that in some respondent groups the IiCs are higher. In this paper, various alternative specifications of the basic model in which this relationship is explicitly specified will be explored and compared with each other. A first approach is a two-step procedure. In the first step the ICC are calculated within the categories of particular respondent characteristic. In the second step, these ICCs are used as dependent variable in a model with respondent characteristics as independent variable. In a second approach a multilevel model is specified in which the variance of the residual errors at the respondent level and the variance of the random intercept at the interviewer level are calculated conditional on the categories of a respondent. With this specification it is possible the calculate conditional ICCs. In the third approach the ‘mixed effects location-scale model’ is used. This model is an elaboration of the standard two level random intercept model and allows to evaluate the interviewers’ impact not only on the means (standard model) but also on the variability of the respondents’ answers. This means that an additional random effect for the residual variance at the respondent level is specified.