Mixed modes & mode effects 2 |
|
| Chair | Ms Franziska Gebhard (German Internet Panel, SFB 884, University of Mannheim, Germany ) |
There is broad evidence from the field of survey methodology that empirical results differ depending on the respective mode and response scales. Thus, my research examines re-sponse scale effects for different modes on responses and data quality in surveys about polit-ical attitudes towards security and defense policy, which are assumed to be sensitive topics.
To examine these measurement effects, a large-scale method experiment was conducted in two concurrent surveys (CATI and CAPI) with probability samples of the German population. Both surveys included the same 44 items with varying content, stimuli, response labels and number of response options (split-ballot: 4-, 5- and 7-point scale). To analyze interaction ef-fects of the response scale format in both modes with interviewees, various characteristics were documented. To get more information about the interviewee and the interview situation, the interviewer was asked to assess respondents’ characteristics and behavior during the in-terview and the interview situation in general. In addition, metadata (response latency, day and time of interview) were recorded. To control for interaction with interviewer characteris-tics, socio-demographics and job experience were included in the dataset.
In the presentation, empirical results are presented and discussed.
With the continuous reduction of face to face and telephone response rates and growing costs, interviewer-administered surveys face an increasing competition from online mode. Being more cost effective, it is also free of issues associated with the interviewer-mediated data collection, such as interviewers’ effect or social desirability bias. However, in online surveys other points of concerns emerge. While some may be controlled for by survey design, others such as coverage or non-response bias are difficult to resolve.
The Fourth European Quality of Life Survey conducted between September 2016 and February 2017 in 28 EU Member States and 5 EU candidate countries, is commissioned by Eurofound and managed by Kantar Public Brussels. It provides a fine source of information about the mode effects in terms of modes coverage, response bias and online non-response. In the study, apart from the main data collection mode of face-to-face survey, in four countries (Germany, Poland, Slovenia and United Kingdom), a follow-up online survey was conducted covering, among others, a number of questions included in the face-to-face stage of the study. The target group of the follow-up survey were a) respondents of the mainstage survey who had been online at least once in the last 12 months, and b) the non-contacts from the mainstage survey.
In our analysis, we differentiate four groups of face-to-face mode respondents: (1) web ineligible – those who stated they do not use the Internet; (2) web refusals – those who refused to take part in the follow-up study; (3) web non-respondents – those who agreed to take part in the survey but have not completed it and (4) web respondents – those who took part in the online survey. We perform a series of comparisons between those groups, looking both at the sociodemographic profiles, as well as responses on substantive questions, in search of differences that correlate with the response questions and thus in turn, if not controlled for, may lead to biased results.
We will compare group (1) with groups (2), (3), (4), analysing differences between their socio-demographic characteristics, which will show the coverage bias of the online follow up of a face to face survey. Additionally, we will analyse their survey responses, providing insight into magnitude and nature of the online mode bias. We will compare groups (2), (3) and (4) in search of significant patterns between the web survey respondents and the non-respondents, in their face to face survey responses.
Response distribution and data quality are influenced by different mode of data collection and measurement design. One commonly examined consequence on data quality is social desirability bias, which is related to how survey questions are measured, whether an interviewer is involved, pace of cognition process during interviews, sensitivity of survey questions, etc. In addition to sensitive questions, values that are societally or culturally approval or accepted often cause social desirability bias in self-reported response. For example, happiness is often overestimated due to its high priority in some societies. Cross-national studies have indicated that more than 70 percent of the respondents in most of the countries reported high level of happiness. However, it is possible that “being happy” is a socially desirable value or a goal of national development. The results of happiness, however, has not been fully examined from the perspective of social desirability using empirical data.
This study aims address this important issue by using survey data collected from face-to-face interviews (CAPI) and telephone interviews (CATI). Previous studies often found that surveys with self-completion are less likely to have social desirable response when compared to those involved with interviewers. However, it is not clear whether the level of social desirability will be different between telephone and face-to-face interviews, where interviewers are present in both modes of data collection. It is even less common to examined mode effect for a non-sensitive issue, such as happiness, when social desirability bias is considered. The association between social desirability and happiness will then be included in analysis. Since social desirability can be seen as a latent construct, structural equation modeling will be used for analysis. The findings are expected to contribute to the field of survey methodology as well as that of subjective well-being.
The French Labour Force Survey will have to evolve, in particular following the adoption of the future European Regulation on new, integrated ways to collect and use data from social surveys, and more generally in the context of reorganising the production of European statistics on households (Integrated European Social Statistics - IESS). The developments regarding the “at work” sub-module could lead to structural breaks in time series of the main labour market indicators (eg unemployment and employment rates). Eurostat called for application to lead an experimental household survey to increase knowledge on the questions “at work” of this new module. Another goal was to be able to get all the forms of employment (including small jobs or students' jobs) in the Labour Force Survey. The French National Institute of Statistics and Economic Studies (INSEE), currently involved in the task force which develop the model of European questionnaire, volunteered to test the questionnaire. Furthermore INSEE was interested in this experimentation to improve its knowledge in the field of Internet data collection and mixed-mode household surveys.
Following this commitment, INSEE launched an Internet experimental household survey, whose main goal was to evaluate a “questionnaire effect” on the employment and unemployment rates, and, more specifically the effect of one of the variants on the "at work" sub-module. Furthermore, this experimental household survey allowed to get first estimates of “mode effects” (and selection effects) by comparing data collected from the current telephone and face-to-face Labour Force Survey to the results obtained online.
INSEE led an experimental household survey on 40,000 households in metropolitan France from May 23rd till June 26th, 2016. Half of the surveyed households received the questionnaire of the French Labour Force Survey transposed on the Internet ; the other households received one of the alternative versions of questionnaire suggested by Eurostat. INSEE choose the version further from current French Labour Force Survey. Considering the objectives, the questionnaire was restricted to the modules concerning accommodation, position in the labour market, from which the main indicators of the investigation can be estimated, and job's description. Furthermore, the sampling design was established to minimize the variance of the estimates of employment and unemployment rates.
This experimental survey provides first reliable and innovative estimates of “questionnaire effects” and “mode effects” on the main indicators of the French Labour Force Survey. This experimental survey also allowed to pursue the works relative to the aggregation of the data collected in the field and data collected by Internet. Beside this main question, it is necessary to think about nonresponse factors and quality of collected data.
It is argued that traditional ways of collecting social surveys are threatened by the rising data-collection costs and the declining response rates. In an attempt to solve this problem, researchers have started to utilize cheaper and easier data collection methods, especially those focusing on various types of online data. Current research on survey methodology has criticized sample-to-population representativeness of many online surveys.
At the same time, however, research on how the mode of data collection affects to responses is almost completely lacking. In this article, we analyze differences between response modes by examining questions pertaining to immigration. If attitudes toward immigration are different between users of internet surveys and paper surveys, then the prevalence of combined data collection may distort the data sets used in scientific studies.
Our data are derived from the International Social Survey Program (ISSP) 2013. We selected two countries for the analysis, Finland (n=1, 243) and Norway (n=1, 585), both of which applied similar methods of data collection technique (self-conducted mail survey and web survey).
We found that respondents’ tend to answer more negatively towards immigration via mail-questionnaire than Web-questionnaire. The results indicate also that although the popularity of the Web-surveys has increased during recent years, the mode of response is still associated with socio-demographic variables which are apparent in use of digital technology. However, in conclusion we suggest that the mixed-mode survey is a reliable method of data collection especially after controlling for background variables and their interactions with the response mode.