Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Mode Effects in Mixed-Mode Surveys: Prevention, Diagnostics, and Adjustment 1 |
|
| Convenor | Professor Edith De Leeuw (Utrecht University) |
| Coordinator 1 | Professor Don Dillman (Washington State University) |
| Coordinator 2 | Dr Barry Schouten (Statistics Netherlands) |
Mixed-mode surveys have become a necessity in many fields. Growing nonresponse in all survey modes is forcing researchers to use a combination of methods to reach an acceptable response. Coverage issues both in Internet and telephone surveys make it necessary to adopt a mixed-mode approach. Furthermore, in international and cross-cultural surveys, differential coverage patterns and survey traditions across countries make a mixed-mode design inevitable.
From a total survey error perspective a mixed-mode design is attractive, as it is offering reduced coverage error and nonresponse error at affordable costs. However, measurement error may be increased when using more than one mode. This could be caused by mode inherent effects (e.g., absence or presence of interviewers) or by question format effects, as often different questionnaires are used for different modes.
In the literature, two kinds of approaches can be distinguished, aimed at either reducing mode effects in the design of the study or adjusting for mode effects in the analysis phase. Both approaches are important and should complement each other. The aim is to bring researchers from both approaches together to exchange ideas and results.
This session invites presentations that investigate how different sources of survey errors interact and combine in mixed mode surveys. We particularly invite presentations that discuss how different survey errors can be reduced (prevented) or adjusted for (corrected). We encourage empirical studies based on mixed-mode experiments or pilots. We especially encourage papers that attempt to generalize results to overall recommendations and methods for mixed-mode surveys.
Note: Depending on the number of high quality paper proposals we could organize one or more sessions.
Note 2: We have four organizers, this does not fit the form. Fourth is Joop Hox Utrecht University, j.hox@uu.nl
In recent years, different attempts have been made at emulating dialogue features from interviewer administered modes in web modes. This is partly inspired by grounding theory, which stresses that a common understanding of key concepts is a premise for a functioning conversation. The interview is regarded as a conversational and collaborative effort rather than as rigidly standardized script.
The Norwegian Labour Force Survey is currently a CATI only survey. Qualitative tests show that for certain groups of respondents, interviewers utilize conversational interviewing and provide different types of clarifying information and calculations, to obtain valid responses.
Within the framework of the ESSnet project Data Collection for Social Surveys using Multiple Methods (DCSS), Statistics Norway is making a CAWI questionnaire for the Labour Force Survey, to be user tested in the spring of 2013. In an effort to prevent mode effects, we will introduce interviewer-emulating dialogue: Clarifying information on key concepts will be available through hyperlinks. "Don't know" responses will trigger a probe asking what made the question difficult to answer, followed by help targeting that/those problem(s). Calculators for contracted and actual working hours, tailored to different working time arrangements, will be offered.
The paper will present results from cognitive and usability tests on different devices ranging from PCs to mobile phones, including eye tracking. The paper will also present results from a small-scale pilot survey, on how the dialogue/help features were used. In conclusion, the paper will discuss how the questionnaire should be developed further
The BIBB/BAuA-Employee survey of 2011/2012 on behalf of the Federal Institute of Vocational Education and Training (BIBB) and the Federal Institute for Occupational Safety and Health (BAuA) included data collection in three modes, thus allowing the systematic comparison of the measurement of identical concepts concerning the working conditions in CATI, CAPI and CAWI. About 20.000 employees were interviewed per telephone, about 2.100 in face-to-face and about 2.400 online. The questionnaire was designed to measure the same concepts in all three modes.
Analyses show differences as well as similarities in the results between the three modes: Bi- and multivariate analyses give hints on interesting mode effects concerning non-differentiation, social desirability and acquiescence. Also, differences between the various modes regarding the level of composite indices can be found. On the other hand, confirmatory factor analyses and structural equation models indicate equivalence of the measurement in the three modes. That is, variance and covariance structures seem to be similar.
The results give helpful hints for the design and the implementation of mixed-mode surveys. This concerns the formulation of questions and the design of the questionnaire as well as issues to be considered in the phase of the analyses of surveys.
Data collection mode may influence survey results. Examples of mode effects include the type of communication channel and the presence of an in-terviewer.
One considerable difference between CATI and CAWI is the don't know (DK) options. In our study, we try to find an equivalent way to present the DK option between CATI and CAWI questionnaires. The broader objective is to develop mixed-mode data collection for the Labour Force Survey (LFS).
In the Finnish LFS CATI questionnaire, the DK option is available for interviewers but they never offer it directly. Therefore, the presence and layout of the DK option in CAWI questionnaire has also been designed from that perspective.
By qualitative pre-testing (25-30 respondents), we study whether respondents need the DK option and if so, do they visually notice it while answering the CAWI questionnaire? This study will be completed by the end of April 2013.
In addition to qualitative pre-testing, we also carry out a web pilot study (a random sample of 6,000 Finns aged 16-64). In this study, we will have a split-ballot experiment to compare two different ways to present the DK option. Furthermore, the distribution of DK answers will be compared with the LFS CATI data collected at the same time. The data collection of the pilot study takes place in May 2013 and preliminary results will be ready in summer 2013.
The project is part of the Eurostat ESSnet project Data