Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Explaining Interviewer Effects in Interviewer-Mediated Surveys 1 |
|
| Convenor | Professor Annelies Blom (University of Mannheim) |
| Coordinator 1 | Ms Julie Korbmacher (Munich Center for the Economics of Aging (MEA), Max Planck Institute for Social Law and Social Policy) |
Researchers are invited to submit proposals for papers at the session "Explaining Interviewer Effects in Interviewer-Mediated Surveys" at the European Survey Research Association conference, July, 15-19, 2013 in Ljubljana. In interviewer-mediated surveys interviewers naturally have great potential to affect data quality. Interviewers compile sampling frames; they make contact and gain cooperation with the sample unit; and they act as mediators between the researcher's questions and the respondent's answers. Their characteristics, attitudes, experience and abilities can affect all stages of the data collection process and interviewer effects may occur. As such they are invaluable and a source of error at the same time.
Therefore, the selection of good interviewers and appropriate training are essential for high-quality surveys. However, still little is known about what constitutes a good interviewer and good training. Understanding the mechanism of interviewer effects requires the availability of information about the interviewers. There are three potential sources of interviewer information: First, the actual interview data and information contained therein about interviewer clustering. Second, paradata automatically collected during the data collection process. This may include information about how the data were collected (e.g. call record data), as well as information on the interview itself (e.g. response times or audio trails). Third, a survey administered to the deployed interviewers may collect data about relevant interviewer characteristics. Such a survey may cover experiences, attitudes, expectations, and general demographics.
This session will focus on research into explaining interviewer effects on various aspects of a survey using one or more sources of information about the interviewer.
There is evidence that interviewers (Matschinger et al. 2005) as well as respondents (Kreuter et al. 2011) may be tempted to answer filter questions in a way that minimizes the number of follow-up questions. This may become relevant when data on ego-centered networks are collected (van Tilburg 1998, Marsden 2003). The number of persons reported on a network generator question can have a huge impact on subsequent interview duration if multiple follow-up questions on each alter or even pairs of alters are asked.
We analyze interviewer effects on a network size filter question in the PASS panel. Here five follow-up questions were only asked for the three closest friends. However, if network size was reported to be smaller than three, up to 15 questions could be skipped.
As a mixed-mode survey (CATI/CAPI) PASS is well suited for the investigation of interviewer effects. Applying multilevel models we find no interviewer effects in CATI mode where interviewers were paid by the hour and frequently supervised. In CAPI, however, where interviewers were paid by case and not closely supervised, incentives to cheat are considerable. Here we find a strong interviewer effect on network size. As the area-specific network size is known from telephone mode (where allocation to interviewers is random), interviewer effects and area effects can be seperated.
While we find no general trend that network size decreases by interviewer ecperience, a latent class regression analysis shows that subgroups of interviewers learn cheating during the fieldwork.
It is well known that interviewer behavior affects data quality. It is less well known, however, that certain types of interviewer behavior might even "destroy" survey data. If interviewers are paid per interview completed, they have strong incentives to shorten the interview. This can be done for instance by answering filter questions so as to skip lengthy follow-up questions. If many interviewers show such "rational" behavior data quality will be damaged seriously.
We provide evidence from a large German survey (German Family Panel) that such rational interviewer behavior is indeed widespread. In particular, we find that ego-centered networks generated in waves 2 and 4 of the Family Panel are surprisingly small. We also find large interviewer effects which cannot be explained by interviewer or respondent characteristics. We try to identify those interviewers who are particularly responsible for the large interviewer effects by using the jackknife. We identify three groups of interviewers: Those who elicit very small networks ("fraudulent" interviewers), those who generate particularly large networks ("diligent" interviewers), and the rest ("normal" interviewers). Further we show that only the diligent interviewers provide networks of a reasonable size. In addition, substantial results obtained with network data from normal/fraudulent interviewers differ from those obtained with data from the diligent interviewers.
Because it seems difficult to change the incentives of interviewers in large-scale face-to-face surveys (at least in Germany), we conclude that one has to be very cautious, when analyzing network data obtained from such a survey.
This paper examines the determinants of consent to the collection of biomarkers in SHARE with special regards to the role of the interviewer administering the collection.
The Survey of Health, Ageing and Retirement in Europe (SHARE) expanded measurements of objective health by collecting a battery of innovative biomarkers. A new module was implemented in the fourth wave of the German SHARE Study which included the collection of dried blood spots. For this measurement the ethics review board requires the respondents' written consent. The interviewer plays an important role in the collection of biomarkers: (s)he is not only responsible for explaining the measurements and reassuring respondents, but is also the one conducting all measurements. Especially in the case of dried blood spots a high level of interviewer skills and trust of the respondent in the interviewers' abilities is necessary. As the interviewer plays such a crucial role in the collection, we examine their influence in this work. Information on them can be drawn from the 2011 interviewer survey of the German SHARE interviewers. This PAPI questionnaire was administered during field training and includes information on general attitudes towards surveys as well as some questions on interviewers' attitudes, experiences, and expectations with regard to the collection of biomarkers. Effects of these variables on the consent rates of the interviewers will be investigated.
In the context of the German Socio-Economic Panel, longitudinal survey data exists not only on respondents but also on interviewers. In the years 2006 and 2012, interviewers filled in a short version of the standard SOEP questionnaire.
This research design allows us to match attitude change in respondents with attitude change in the same domain in their interviewers. Moreover, the measurement of personality traits as well as cognitive abilities in interviewees and interviewers as well as the measurement of interviewers' job satisfaction and expectations permits us to shed light on the sources of the prevalence of interviewer effects.
Building on a review of the psychological survey literature, we distinguish between two sources of interviewer effects: the first is mediated via a cognitive route and represents a type of informational influences; the second source is the pressure of conformity mediated via an affective route. We empirically identify interview settings that are more or less prone to one of these two influences and provide guidance for personalized interviewer training.
The analysis suggests that informational influences are most likely to occur if respondents' involvement in the topic is low as well as their cognitive abilities. Interviewers' informational influence is expected to increase with increasing job experience, cognitive abilities, as well as levels of extraversion and conscientiousness. We further suggest that conformity pressure is most likely to affect response behavior if respondents score high on the personality traits of neuroticism and agreeableness and if they possess a high level of interpersonal trust.