Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Is it worth mixing modes? New evidence on costs and survey error on mixed-modes surveys 1 |
|
| Chair | Dr Ana Villar (City University London) |
Mixed mode surveying has been driven by two targets: cost reduction and reduction of non-response bias. Costs can only be reduced if interviews are shifted from expensive interviewer-administered interviews to less expensive self-administered interviews. When costs are not the primary reason mixed mode techniques will win with the idea to offer many interview modes and leaving the choice of the mode to the interviewee to increase participation rates.
We will show realistic calculation examples for various common combinations of mixed mode surveys (e.g. CATI/CAWI, CATI/CAWI/PAPI, etc.). Costs for program and data preparation depend on the survey software used and if mixed modes surveys are supported by the following features: integrated data base, single programing environment for all modes, reporting across all modes, question wording by mode, etc..
We will also outline how additional options contribute to the desired outcome of a maximum participation rate and what costs have to be considered. We find inbound calls (the interviewee calls the agency actively to conduct the survey), invitation letters and invitation SMS important and promising features within mixed mode survey approaches, as well as the integration of mobile devices across all survey modes. Additional costs arise from designing questionnaires for these multiple devices.
Our experience with mixed mode surveying comes from projects conducted for Swiss federal statistical office and from customized market research studies.
Mixed mode data collection is gaining popularity in survey research internationally as a potential solution to the problem of declining response rates, rising fieldwork costs, and undercoverage in telephone surveys. This makes it particularly appealing in Switzerland where response rates are generally low, data collection budgets are high, and where telephone interviewing remains the preferred mode of conducting surveys. Because of this, mixed mode solutions for surveys where telephone is the primary mode are fast becoming common practice, yet until now, little empirical evidence has been available about the real benefits and costs of different mixed mode designs in the Swiss context. We present the results of a new mode comparison study designed for this purpose. A probability sample (from the population register) of 3600 adults living in the French-speaking region, was randomly assigned to one of three mode treatment groups (CATI, web, or mail, depending on the availability of a registered telephone number) to complete a questionnaire about personal and social wellbeing. Sequential mixed mode designs were tested in the CATI and web groups, consisting of a postal follow-up in both groups, followed by CATI or face-to-face follow-ups of nonrespondents in the web group. We present initial findings relating to the impact of mode on coverage and nonresponse errors using socio-demographic data from the sampling frame, while at the same time evaluating the relative costs associated with different data collection designs, and the different stages of data collection designed to reduce nonresponse rates.
The target population of the Berlin Aging Study II (BASE-II) consists of 1600 persons aged 60 to 80 and of 600 individuals aged 20-35. A survey on the social situation and attitudes is conducted by the SOEP Group at the DIW Berlin using a mixture of web and mail interviews. However, quality concerns raise doubts of whether web interviews among the elderly are of the same quality as the mail interviews. The internet represents the fastest communication medium and information is processed more superficial if presented through this channel. This problem is expected to occur in particular among the elderly, a population that is less familiar with the internet.
To test the mode effect on total survey error in the elderly and the reference group of young respondents, we conducted an experiment in which the choice of survey mode was manipulated as we offered monetary incentives that varied in their amount conditional to the choice of data collection mode. It turned out that the web/mail rate differed by 50 percentage points due to the manipulation.
Using an instrumental variable approach, we investigate the mode effect on total survey error using the randomized monetary incentives as exogenous variation in the survey mode. We focus on mode effects in unit and item non-response, the prevalence of response styles, and the reliability of multi-item measures. We also discuss the question, whether or not the possibly reduced data quality of web interviews in the elderly outweighs the reduced costs.
In the context of the German Health Update (GEDA) we conducted a methodological survey in order to compare response rates and sample structures of two different mixed-mode designs including Web, PAPI, and CATI modes of data collection. The aim was to evaluate the most appropriate design according to costs, representativeness and data-quality. In the following presentation we want to outline and discuss the results from the web modes - once implemented in a parallel mixed-mode design and once implemented in a sequential mixed-mode design.
Subjects were randomly selected from official registers in Germany. Six sample points were chosen considering urban and rural as well as western and eastern regions. The sample was stratified according to age and sex. We sent written invitations to subjects including a link and a personal PIN and asked them to fill in a questionnaire from the health survey. Half of the individuals were invited to take part in the sequential design, the other half were asked to take part in the parallel design. Our preliminary results show that significantly more people tended to fill in a web questionnaire when this mode is offered first (sequential design). When people were invited to choose between Web, PAPI or CATI mode from the beginning, the response for PAPI was remarkably higher and less than one fifth of the respondents chose the web mode. Further results regarding the demographic characteristics of the respondents and the sample composition will also be presented.