Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Mode Effects in Mixed-Mode Surveys: Prevention, Diagnostics, and Adjustment 3 |
|
| Convenor | Professor Edith De Leeuw (Utrecht University) |
| Coordinator 1 | Professor Don Dillman (Washington State University) |
| Coordinator 2 | Dr Barry Schouten (Statistics Netherlands) |
Mixed-mode surveys have become a necessity in many fields. Growing nonresponse in all survey modes is forcing researchers to use a combination of methods to reach an acceptable response. Coverage issues both in Internet and telephone surveys make it necessary to adopt a mixed-mode approach. Furthermore, in international and cross-cultural surveys, differential coverage patterns and survey traditions across countries make a mixed-mode design inevitable.
From a total survey error perspective a mixed-mode design is attractive, as it is offering reduced coverage error and nonresponse error at affordable costs. However, measurement error may be increased when using more than one mode. This could be caused by mode inherent effects (e.g., absence or presence of interviewers) or by question format effects, as often different questionnaires are used for different modes.
In the literature, two kinds of approaches can be distinguished, aimed at either reducing mode effects in the design of the study or adjusting for mode effects in the analysis phase. Both approaches are important and should complement each other. The aim is to bring researchers from both approaches together to exchange ideas and results.
This session invites presentations that investigate how different sources of survey errors interact and combine in mixed mode surveys. We particularly invite presentations that discuss how different survey errors can be reduced (prevented) or adjusted for (corrected). We encourage empirical studies based on mixed-mode experiments or pilots. We especially encourage papers that attempt to generalize results to overall recommendations and methods for mixed-mode surveys.
Note: Depending on the number of high quality paper proposals we could organize one or more sessions.
Note 2: We have four organizers, this does not fit the form. Fourth is Joop Hox Utrecht University, j.hox@uu.nl
Collecting data through mixed-modes is becoming an ever more popular and necessary part of survey design, impacting both cross-sectional and longitudinal data. While interest in the topic is longstanding, our knowledge of its effects on measurement quality in panel studies is limited. The present paper will contribute to this debate by analyzing the impact of mixed modes of data collection on reliability in the Innovation Panel. This is a subsample of Understanding Society, a household panel survey representative of the UK population, that carries out methodological experiments. The random assignment to mixed mode (CAPI versus two types of CATI) in the second wave will give the opportunity to see the impact of mode design on measurement quality in this wave and subsequent ones. To assess these effects I will estimate the reliability of a subset of items over four waves using quasi simplex models for the different mode designs. I will further explain the differences in reliability by modes through variable level characteristics such as question complexity, topic or level of measurement.
To reduce nonresponse and coverage error at affordable costs, mixed-mode surveys are often advocated. The final goal of mixed modes is combining data from different sources, which assumes that data can be validly combined. Online surveys are a relatively recent tool for data collection and as a result mode comparisons with online surveys are still scarce. In this paper we explicitly investigate error structures for a combined online and telephone survey. This popular mixed-mode design is not altogether unproblematic from a measurement error perspective, due to differences in audio and visual channels of communication.
When attitudes or other subjective phenomena are studied, hard validation criteria (e.g., record checks) are not available. We therefore used a direct model-based approach (multitrait-multimethod or MTMM) for the analysis of the error structure in this study. The MTMM model allows for a statistical separation of substantive or trait variance, method variance, and random or error variance. The data were collected using the Dutch LISS-panel, which is a high quality, probability-based Internet panel. Panel members were randomly assigned to one of two modes: a computer assisted telephone interview or a web survey. Mode of data collection is the Method factor and within each mode the same five questions (Traits) were posed. One month later the same respondents were asked the same questions, but now in a uni-mode (web) survey. This design enables us to disentangle systematic and random error in both telephone and web survey and to investigate
Sequential mixed-mode surveys combine different data collection modes in succession to reduce nonresponse bias under certain cost constraints. However, as a result of nonignorable mode effects, nonrandom mixes of modes may yield unknown bias properties for population estimates such as means, proportions and totals. The existing inference methods for sequential mixed-mode surveys generally assume that mode effects are ignorable. The objective of this paper is to describe and empirically evaluate some proposed multiple imputation estimation methods that account for both nonresponse and nonrandom mixtures of modes in a sequential mixed-mode survey. In particular, the multiple selection imputation models allow imputation of responses for alternative modes as if they responded in a given mode by controlling nonrandom mixes of mode. For example, if personal and telephone interviews are used, one step in the process is to impute values for the telephone cases as if they had responded by personal interview (PI) to produce a completed PI data set. Similarly, a completed telephone interview (TI) data set is created. The completed PI and TI data sets are combined for inference. Through simulations, the method is evaluated in terms of the bias reduction for varying degrees of mode effects and model fit. The American Community Survey (ACS) or the 1973 public-use Current Population Survey and Social Security Records Exact Match data will be used to conduct empirical and simulation evaluations. The focus of the empirical evaluations and simulations will be mean family income.
Mixed-mode surveys are frequently used to improve the survey participation but statistical tools for analyzing mixed-mode survey data are relatively underdeveloped. Motivated by a real survey in Korea, we consider an imputation approach to handling mixed-mode surveys. The proposed method uses measurement error models to explain the mode effects and then imputation is to predict the counterfactual potential outcome in the measurement error model. The model parameters are estimated using the method of parametric fractional imputation of Kim (2011). The proposed method is applied to the survey of private education expenses in Korea.