Mixed modes & mode effects 1 |
|
| Chair | Professor Franciszek Sztabinski (Institute of Philosophy and Sociology Polish Academy of Sciences ) |
| Coordinator 1 | Professor Pawel Sztabinski (Institute of Philosophy and Sociology Polish Academy of Sciences) |
Standardised surveys are an important tool for electoral research. The increasing usage of CAWI mode and mixed mode raise the question, if different modes lead to different substantial findings.
Comparing surveys conducted in different modes (CAPI, CATI, CAWI) and mixed mode surveys (CATI/CAWI-mix), we find that simple statistical measures (distributions, accurateness of prediction,...) differ.
We test the hypothesis, that CATI surveys are more biased towards politically interested respondents, as survey participation is more likely if someone is interested in the topic. This has implications for the accurateness of predicting and explaining voter turnout, but also vote choice for Austria's radical right populist party FPÖ and other parties.
Moreover we address the issue of social desirability bias, again for voter turnout and vote choice for the FPÖ.
When comparing results of electoral choice models, we find that overall the results do not differ substantially.
As CATI and CAPI surveys are usually based on probabilistic samples but samples for CAWI surveys are usually drawn from access pools, questions about sample strategies occur.
We use the results to come up with conclusions for practitioners and academics for designing electoral survey within restricted budgets.
We use data from Austria, in particular CATI, CAPI and CAWI surveys conducted before and after the Austrian federal election 2013 by the Austrian National Election Study AUTNES, and mixed mode surveys conducted by IFES before and after the presidential elections 2016. If another federal election is called in 2017, we will include survey data for this election as well.
The procedure used to collect survey data has a significant impact on aspects such as the cost of the survey or the time needed to complete the study. But the data collection method considered also may influence the responses provided by the respondents and, consequently, the results of the survey.
Face-to-face interviews were one of the first methods used to gather survey data. In a face-to-face interview, an interviewer personally meets the people selected in the sample in order to obtain their responses to the questions of a questionnaire previously designed. The presence of the interviewer may be very useful for clarifying questions or motivating the respondents to complete the survey. Nevertheless, interviewers are also likely to cause, directly or indirectly, biased responses from the surveyed people, especially when asking questions about sensitive aspects. Usually, respondents provide the responses that are “socially well-accepted” when asked for sensitive or intimate aspects of their life in presence of other people, even if these responses are not true. This behaviour, called social desirability, leads to incorrect results and so, to wrong conclusions.
Self-administered approaches for data collection reduce the problem of social desirability since they do not involve any interviewer. In these situations, respondents usually have a higher perception of anonymity, which leads them to provide truthful answers. In addition, self-administered procedures are cheaper and faster to apply compared to face-to-face interviews, so, they seem appropriate to collect sensitive data. However, response rates are, generally, quite low when using self-administered techniques since respondents tend to abandon the survey prematurely, especially when the questionnaire is long or when the questions are complex. This could affect to the quality of the data. Among self-administered methods for data collection, web surveys are the most used one at present. Indeed, web surveys have gained much popularity among the scientific community due to the fast development of the technology and the increase of the Internet penetration all over the world. But in some countries, access to the Internet is still limited, which may result in serious undercoverage biases when using this type of surveys.
In this work we compare the quality of the responses to questions related with addictive behaviours collected through face-to-face interviews and through a web survey. We measure the global response rate for each method as well as the particular response rate for each question to detect differences depending on the type of question and the method considered for collecting the data. Then, we analyse the answers provided by the respondents to identify and compare the response patterns of each group of individuals.
Moving existing establishment surveys from face to face, mail and telephone to the web can make data collection cheaper and more efficient and allow the use of techniques like dependent interviewing, plausibly checks and complex filters. However, there is little research into the effect of switching to web data collection on data quality. We conducted a survey of establishments in Germany which experimentally varied three mode conditions: web only, mail only and a web/mail choice condition. We also compare the unit and item nonresponse rates across the conditions. To explore unit nonresponse we compare the proportion of initial and respondent sample by establishment size and industry. In order to understand how the mode affects item nonresponse we run multiple regression models. We repeat our model over all questionnaire sections and over certain question formats and types. Our results show that the web has a lower response rate while the mixed web/mail choice condition performs as well as the mail mode. Our research has also shown that participating via web (independent of the mode condition) can reduce the overall item nonresponse rate. Our results are relevant for establishment surveys that wish to move to the web, but are worried about and nonresponse effects on the data.
In France, the census is placed under State responsibility.
Since 2004, the census is based on annual data collection, successively surveying all the municipal territories over a five-year period.
The conduct of census surveys is based on a close partnership between the municipalities and INSEE.
Since 2015, it is possible to be enumerated by internet: an enumerator recruited by the municipality offers to households the possibility to fill in a web questionnaire, or a paper one.
The proposal of differents modes is up to the municipality and the enumerators, so the web response rate is very variable between municipalities.
In order to explain the mode preference and analyze mode effects, we better have to control the « enumerator » or municipality effect, if existing.
We modelize the propensity to choose to answer by web, using demographic caracteristics of the household and its members, taking into account information on enumerator in a multilevel model.
After matching respondents from both modes, we analyze the effect of the new mode, especially on non response, and on the distinction between permanent residents and non permanent ones.
As the census questionnaire is self-administered in both modes, we can expect small mode effects.
But the web questionnaire allows to implement controls and probes after non response. It leads to lower partial non response for web questionnaires.
The web questionnaire contains an automatic classification of every person as permanent or non permanent resident.
Children studying far from their parent's home should be counted apart. We can compare the part of students well classified in both modes, depending on the distance between their parents' city and the location of the educational institution.
As the web response rate is growing, we want to take advantage of interactive possibilities while ensuring data quality, and consistency between modes.
The Mixed Mode design is intended to improve the response rate and reduce the nonresponse error in comparison with the unimode design and, at the same time, it is expected to reduce the overall survey costs (de Leeuw, 2005; de Leeuw, Hox, and Dillman 2008; Dillman, Smyth, and Christian 2014). However, methodological studies which aspire to improve this design usually focus on using the cheapest modes, i.e. mail and web surveys. Little is known about the use of face-to-face interviews (F2F) as one of the modes in Mixed Mode studies. Meanwhile, the vast majority of important cross-national studies, such as WVS, GSS, ESS, EVS or Eurobarometer, are currently administered in the F2F mode. The use of other, less costly modes alongside F2F interviews could, therefore, boost the achieved sample and reduce the costs of surveys.
Experiments conducted in connection with the European Social Survey (ESS) 2012 and 2015 and the Polish Panel Survey (POLPAN) 2013 indicate that the use of other modes in a follow-up survey (with F2F as the initial mode) increases the response rate only very slightly. With this in mind, another experiment was conducted in 2015, where the reverse order was applied, i.e. a self-administered questionnaire (mail and web-based) was used as an initial mode, with F2F interviews used as the follow-up mode. The experiment was based on the ESS 7 questionnaire. The response rate, which stood at 23.6% in the initial phase, rose by 32.3 percentage points following the use of F2F interviews, reaching 55.9%.
The analysis of the sample composition in terms of demographics and substantive variables, known from research to be linked to survey participation (social involvement, political involvement etc.), suggests that the application of the F2F mode in the follow-up survey improved the sample composition in most cases, in comparison with the initial mail-based mode.