Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Is it worth mixing modes? New evidence on costs and survey error on mixed-modes surveys 3 |
|
| Convenor | Dr Ana Villar (City University London) |
| Coordinator 1 | Professor Peter Lynn (University of Essex) |
Survey designers face a continuous tension between minimizing survey error and keeping costs as low as possible (Groves, 1989). One of the strategies that have been pursued to reduce costs is the use of mixed modes of data collection, using cheaper modes early in the process, and reserving more efficient and expensive modes to increase response rates and coverage.
A considerable amount of research has tried to assess the impact of using mixed modes of data collection on data quality in terms of response error and measurement error. This body of research typically compares response distributions and response rates across modes but fail to report the effect of the mode design on actual comprehensive costs and on timeliness.
However, mixed mode survey implementations may not be as efficient as first thought. With the current technological tools, the costs associated with the production of equivalent questionnaires across modes, equivalent contact forms, equivalent data protocols, and other fieldwork documents might be an underestimated burden. Further to this, findings about the effects on response rates and measurement effects are far from conclusive, and the field is in need of new evidence linking total survey error and survey costs.
In this session we invite studies that address challenges and lessons learned from the implementation of mixed mode designs, with an emphasis on the link between survey error and costs. Papers submitted for this session will ideally include evidence of the effect of the use of mixed-modes on:
- costs, time, and other resources;
- coverage error;
- response rates and/or response bias;
- measurement error.
Survey researchers increasingly use mixed-mode surveys for general population data collection because mixed-mode surveys are argued to provide lower selection error relative to single-mode surveys. Nevertheless, the advantage of lower selection error might be counteracted by higher fixed costs and by higher measurement error. Surprisingly, this trade-off between selection error, measurement error, and costs has hardly been studied within the existing mixed-mode literature. This paper discusses a possible procedure for evaluating this trade-off by comparing the performance (mean squared error) of mixed-mode survey designs against single-mode survey designs under the same budget constraints. Such a comparison is further illustrated by real example data stemming from a mixed-mode mail---face-to-face survey. This illustration yields an advantage of single-mode designs under low budgets but an advantage of mixed-mode designs under large budgets. However, the validity of these results depend on several modelling assumptions which may be topics for future research.
The ESSnet project on "Data Collection for Social Surveys using multiple modes" (DCSS) is a project launched by Eurostat and is executed by five European National Statistical Institutes (NSI's). The project started in the autumn of 2012 and covers two major topics: (a) Design of web surveys and (b) Multi-mode data collection.
The first task undertaken in the project is a review of the 'state of the art' of web surveys and mixed mode data collection in statistical institutes. A query will be sent out to all European NSI's, selected other NSI's and other organisations that execute general population surveys. A number of topics in the query cover the subjects that are central to this session: costs of fieldwork versus costs of other organisational disciplines, timeliness issues, response rates of mixed mode versus unimode surveys and measurement errors. The query will be held in the coming months, and results will be available for the conference. The paper will in addition give ample attention to Statistics Netherlands experiences. Most SN social surveys are administered mixed mode, including web. Recent evaluations and experiments have focussed on the issues of costs and quality.
Statistics Finland has tested the mixed-mode of a web and CATI survey for collecting the EU Consumer Survey data. The aim of the pilot survey was to find out what kinds of effects the mixed-mode survey has on response rates, the quality of collected data and data collection costs.
The data for the pilot survey were first collected as a web survey between 31 October and 7 November 2012 and the rest of the data were collected as telephone interviews from 7 to 21 November 2012. Mixed-mode for the survey was piloted once before, in 2011.
In the 2011 pilot survey there was a clear difference in the key variable of the Consumer Survey, the consumer confidence indicator, between web and telephone respondents. The first results in the pilot survey 2012 seem to have a similar trend. The assessments of web respondents on micro- and macroeconomics were more pessimistic than those of telephone respondents.
In the 2012 collection, the web was kept open only during the first data collection week, while in the 2011 pilot project, the web inquiry was open throughout the data collection period. Contrary to what is generally estimated or claimed, mixed-mode did not raise the response rate.
Web collection may lower the data collection costs, e.g. the wage costs of interviewer work. On the other hand, a mixed-mode survey often requires more expert work and the preparation of data collection depends on more people than conventional telephone interviewing.
Mixed-mode surveys are conducted in social research with the intention to avoid non-response, sample attritions and low representativeness. There is no consent concerning the question what kind of mixed-mode design leads to higher response rates and better sample compositions. For this reason we conducted a methodological survey (N=10.080) within the German Health Update and compared response rates and sample structures of two different mixed-mode designs including Web, PAPI, and CATI modes of data collection. These modes were differently offered to respondents. In a sequential design the three modes were proposed one after another. First, respondents were asked to fill in a web questionnaire. Non-respondents were contacted again and proposed to fill out a paper questionnaire. The remaining non-responders were proposed a telephone interview. In the parallel design all modes were offered from the beginning on. Respondents could choose the mode they wanted to be interviewed with. All subjects were randomly allocated to one design. Preliminary results showed that the response rates were higher in the parallel than in the sequential design. Substantially more web questionnaires were returned in the sequential than in the parallel design. Generally, there were no remarkable differences between the realised sub-samples. In the sequential design sample, the age distribution was less distorted and in the sample of the parallel design the sex distribution was more accurate. Further analyses will be still conducted and discussed regarding the sample compositions according to other demographic and health-related characteristics.