All time references are in CEST
Challenges and Consequences of Switching Face-to-Face Population Surveys to Self-Completion (Mixed-Mode) Designs 3 |
|
| Session Organisers |
Dr Joss Roßmann (GESIS - Leibniz Institute for the Social Sciences, Germany) Mr Michael Blohm (GESIS - Leibniz Institute for the Social Sciences, Germany) Ms Caroline Hahn (GESIS - Leibniz Institute for the Social Sciences, Germany) Dr Oshrat Hochman (GESIS - Leibniz Institute for the Social Sciences, Germany) Dr Jan-Lucas Schanze (GESIS - Leibniz Institute for the Social Sciences, Germany) |
| Time | Thursday 20 July, 16:00 - 17:30 |
| Room | U6-09 |
The public health regulations that were introduced across Europe (and beyond) since Spring 2020 due to the Corona pandemic impeded the realization of personal face-to-face survey data collection or made it even temporarily impossible – potentially with long ranging consequences for this data collection method. For this reason, surveys of the general population, or important subgroups of the population such as voters, chose to switch from face-to-face interviewing to a mix of self-completion survey modes to secure survey responses from all groups of the target population (e.g., respondents without Internet access or those unwilling to respond in a certain mode).
The Corona pandemic, thus, strongly accelerated a continuing trend of moving surveys from interviewer-administration to self-completion modes. With respect to the general trend and the sustained consequences of the Coronavirus in particular, we welcome submissions that present insights on the methodological and practical challenges of switching population surveys from face-to-face to (mixes of) self-completion modes, and on the effects of doing so on the quality of the data.
Contributions may cover but are not limited to the following research topics:
- Approaches to overcome methodological and practical challenges of switching modes (e.g., regarding sampling and coverage)
- Conception and implementation of self-completion mixed-mode designs for general population surveys (e.g., adaptations in questionnaire design)
- Experimental testing of survey designs to maintain or enhance response and quality of measurement (e.g., use of conditional and unconditional incentives, split questionnaire designs)
- Consequences of switches to self-completion on data quality and total survey error
- Implications of mixing self-completion data collection modes on measurement invariance and possible remedies
- Impact of switching data collection modes on time series in repeated cross-section and panel population surveys
Keywords: Self-completion mixed-mode surveys, mode switch, total survey error, data quality
Mr Florian Leible (Statistics Austria) - Presenting Author
Mrs Brigitte Salfinger-Pilz (Statistics Austria)
Statistics Austria implemented the additional survey mode “CAWI” (Computer Assisted Web Interviewing) besides “CAPI” (Computer Assisted Personal Interviewing) for the Adult Education Survey 2022/23 (AES). In order to add a CAWI mode to the data collection design for the AES the following steps were undertaken: 1) re-wording phase: question wording and adaptions appropriate for CAWI, (2) testing phase I: cognitive interviews, (3) testing phase II: pre-tests (CAWI/CAPI) including workflow testing and (4) finalization phase: adaptions, analysis and finalization of the survey instrument.
The presentation will focus on the pre-test using a multi-mode data collection (CAPI/CAWI) in a real survey situation. In-depth insights will be presented on the following topics: complete test of the questionnaire, the workflow, the sampling, analysis of mode-effects as well as the official correspondence with the respondents (e.g. invitation letters). Results show, that (1) the younger, higher educated population is more likely to answer in CAWI mode and a pre-incentive and additional reminder letter is needed to improve survey response overall.
Additionally, we will compare survey response of the pre-test with the main survey, highlighting the adapted sampling, communication and incentive strategies.
Dr Jette Schröder (GESIS Leibniz Institute for the Social Sciences) - Presenting Author
Dr Claudia Schmiedeberg (LMU Munich)
Professor Josef Brüderl (LMU Munich)
Dr Christiane Bozoyan (LMU Munich)
With increasing costs of face-to-face surveys more and more studies are considering a change to self-administered survey modes to reduce survey costs. Such a change is especially challenging for panel studies as it might increase panel attrition and the selection bias of the panel.
Due to institutional changes, the German Family Panel pairfam, a panel study of young and middle-aged individuals in Germany, switched from CAPI to self-administered mixed-mode after 13 years/waves. To investigate the effects of this mode change on data quality, an experiment was implemented in wave 14, in which panel members were randomly assigned to two groups. One group (N=1.200) was approached in the usual Face-to-Face mode. The other group (N=6.864) was surveyed following the planned survey design for future waves: Respondents were invited via postal mail to an online survey (computer-assisted web interview: CAWI) and, if they did not participate, received a paper and pencil questionnaire (PAPI).
We analyze the effects of the randomly assigned survey mode on participation as well as respondent characteristics associated with attrition in both mode groups. Compared to cross-sectional studies we have the advantage that we can investigate selectivity of attrition not only for basic socio-demographic variables but also for other characteristics such as personality traits using the information collected in prior panel waves.
Dr Jan-Lucas Schanze (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Mr Michael Blohm (GESIS - Leibniz Institute for the Social Sciences)
Dr Pablo Christmann (GESIS - Leibniz Institute for the Social Sciences)
Dr Tobias Gummer (GESIS - Leibniz Institute for the Social Sciences)
Mr Achim Koch (GESIS - Leibniz Institute for the Social Sciences)
Dr Joss Roßmann (GESIS - Leibniz Institute for the Social Sciences)
Professor Christof Wolf (GESIS - Leibniz Institute for the Social Sciences)
Large-scale social surveys in Germany were forced to switch from the face-to-face mode to self-completion modes (web interviews and paper questionnaires) because of the corona pandemic. Due to this exogenous shock, a deliberate transition or test of self-completion modes like in the 2017 data collection for the European Value Study (EVS) was not possible. The sudden mode switch created a need for survey experiments to learn about important design decisions for surveys in self-completion modes. In our study, we focus on incentive experiments that aim at answering the research question on how to design incentive strategies for future self-completion surveys that achieve high response rates while mitigating negative effects on item nonresponse and sample composition.
The present paper uses experimental data from the German part of the 2017 EVS, the 2020 methodological study in the context of the German General Social Survey (ALLBUS-MS), the German part of the 2021 European Social Survey (ESS), and the 2021 Pre-Election Cross-Section Survey of the German Longitudinal Election Study (GLES). All four survey programs used self-completion modes to collect data and implemented incentive experiments that included various factors: €5 unconditional prepaid incentives, €10 postpaid incentives, and two ways of combining €5 prepaid with €10 postpaid incentives that were announced at different points in the contact protocol (in the first invitation letter or in the second reminder). We compare various survey outcomes across the experimental groups for each of the four survey programs. In addition to response rates, we analyze the completeness of data (share of partial interviews, item nonresponse), and the sample composition regarding socio-demographic variables, as well as attitudinal and behavioral measures. Following those results, we also evaluate the cost-benefit ratio of the different incentive strategies.
Dr Ruxandra Comanaru (European Social Survey, City University) - Presenting Author
Mr Nathan Reece (European Social Survey, City University)
For several decades, it has been getting increasingly more difficult to collect data using face to face. Recently, and particularly during the pandemic, several surveys switched to self-completion, including push-to-web. If in face-to-face questionnaires, selecting a random member of the household to interview is straightforward (the interviewer selects a random adult), in self-completion surveys this becomes a challenge. The mailed-in invitation letter needs therefore to include clear instructions about which household member(s) should fill out the questionnaire.
Previous studies in the UK looked at the rate of compliance with the random sampling instructions. Some suggested that there was a high degree of non-compliance when using the next/ last birthday method. The European Social Survey conducted an experiment in Great Britain, where random sampling was conducted using the next birthday method. Results suggest that the number of incorrect respondents is significantly lower than previously suggested (11.4% vs 30% for households with more than one respondent); we also identified a category of responses that were inconclusive due to missing or incomplete data. We examined this category in depth to provide a typology of inconclusive responses (lack of data; same month of data collection as month of birthday; several adults in the household share the birthday month etc.). We compared the results from the GB ESS experiment with data collected in other European contexts forced to switch to self-completion during the pandemic (Austria, Cyprus, Israel, Latvia, Serbia). We will discuss strategies to mitigate the potentially erroneous selection of the participant in the household, including clear instructions in the invitation letter, reminders at various points in the questionnaire, as well as inclusion of check variables to allow for a thorough verification of the final data.