All time references are in CEST
Split Questionnaire Designs in Social Surveys: Challenges and Solutions |
|
| Session Organisers | Dr Julian Axenfeld (German Institute for Economic Research (DIW Berlin)) Dr Christian Bruch (GESIS Leibniz Institute for the Social Sciences) Professor Christof Wolf (GESIS Leibniz Institute for the Social Sciences) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
In recent years, split questionnaire designs have gained significant importance, primarily due to the need to reduce survey costs and questionnaire length. In these designs, respondents do not receive the complete questionnaire. Instead, specific parts of the questionnaire, known as modules, are randomly assigned to different respondents, and the resulting design-based missing values are subsequently imputed.
Applying split questionnaire designs to social surveys raises four crucial questions:
(1) How should questionnaire splits be designed to yield good response quality and appropriate respondent burden while also ensuring imputations of adequate quality that allow valid analyses? This includes issues such as constructing modules and determining which and how many items (if any) should be presented to all respondents, i.e. the size of the core module.
(2) What are the effects of split questionnaire designs on respondent behavior, such as response rates and respondent burden?
(3) How should imputation be conducted under the conditions of social surveys? This involves, for instance, selecting imputation methods and models suitable for small sample sizes, low correlations between items, many nominally scaled variables, and a large number of variables in a survey dataset that need to be imputed.
(4) How should the (imputed) data that was collected by a split questionnaire design be provided to data users so that they can conduct their analyses?
We invite papers addressing the challenges of split questionnaire designs in survey practice, particularly with regard but not limited to the aspects described above. Contributions from fields beyond social science that tackle the challenges of split questionnaire designs are also welcome.
Keywords: Split questionnaire designs, multiple imputation, missing values
Dr Julian B. Axenfeld (DIW Berlin) - Presenting Author
Dr Christian Bruch (GESIS Leibniz Institute for the Social Sciences)
Professor Christof Wolf (GESIS Leibniz Institute for the Social Sciences)
An increasing number of social science surveys use split questionnaire designs to reduce questionnaire length, presenting only a subset of several questionnaire modules to each respondent while leaving out others. This approach results in large amounts of planned missing data that necessitates imputation. Research shows that imputation is most effective when each module covers various topics. Yet, single-topic modules may sometimes be preferable from a questionnaire-design perspective. A potential alternative from survey practice is using single-topic modules with an extended core module presented to all respondents that includes key items from all topics. This study investigates whether this strategy yields outcomes comparable to mixed-topic modules. Using Monte-Carlo simulations based on the German Internet Panel, we simulate split questionnaire designs, impute the missing data, and calculate estimates based on these data. Findings suggest that while an extended core module improves single-topic module outcomes, it is inferior to randomly allocated mixed-topic modules.
Dr Michael Ochsner (FORS) - Presenting Author
Split questionnaire designs are advocated to reduce survey cost and to address questionnaire length, for example when switching from interviewer to self-completion mode. While, theoretically, missing values by design are not an issue for statistical analysis as these missings are missing completely at random, problems might arise when analysing subsamples or topics that need a large number of observations, such as working couples or discrimination. Another problem might occur when composite indices are used which include questions that are spread over several questionnaire splits. Finally, there is the risk that order effects follow from the fact that the sequence of questions is different across splits.
Multiple imputation offers the possibility to analyse data containing missing values without reducing power too much and at the same time taking the uncertainty of missing values into account but comes with issues of complexity, as the models social scientists use often include nominal or ordinal variables that can lead to convergence problems. In such cases, categories need to be combined, which can affect results. Multiple imputation also cannot account for potential order effects.
Using data from the European Values Study 2017 fielded as a web/paper survey in a matrix design in parallel to face to face and long web/paper designs in Switzerland, we compare different real-life models across designs. We demonstrate how to apply multiple imputation even in complex situations, but also show that the advantage of reducing respondent burden can be outweighed by the complexity in survey fieldwork and data analysis that comes with split-questionnaires. These issues are exacerbated in cross-cultural contexts, as comparative analyses using data from Germany, Iceland Switzerland show. We provide guidelines on when a split questionnaire can be a valuable option and when caution is advised.
Ms Saskia Bartholomäus (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Adaptive Survey Designs that use respondents' topic interests can reduce the overrepresentation of politically engaged respondents in political science surveys. Politically disengaged respondents would receive a questionnaire combining political and non-political question modules to boost participation, while politically engaged respondents receive a purely political questionnaire. However, assigning respondents based on their political interest to different question modules may distort research if variables of interest correlate with political interest. Instead, researchers could assign only a certain percentage of respondents with low political interest to tailored questionnaires and use missing data procedures based on the main political question module to correct biased estimates. This paper aims to assess whether Adaptive Survey Designs that rely on content variation bias substantive research (RQ1) and whether missing data procedures mitigate this bias (RQ2). Using the probability-based mixed-mode GESIS Panel pop, I simulate several datasets in which 50% to 100% of politically disengaged respondents’ responses to various variables are set as missing. I then run several regression models using the original and simulated datasets to answer RQ1. To answer RQ2, I run the same regression models using inverse probability weights and multiple imputation. Preliminary results suggest that Adaptive Survey Designs that vary a questionnaires’ content can bias substantive research if more than 50% of respondents are randomly assigned to question modules that are supposed to be more interesting to them. Applying missing data strategies reduces biased estimates and corrects confidence intervals, however. I am going to discuss the potential and risks of the proposed method.
Mr Thomas Merly-Alpa (Insee) - Presenting Author
In 2023 and 2024, Insee, the French National Institute of Statistics and Economic Studies, conducted its national survey on housing conditions. For the first time, this survey employed a mixed-mode design using a sequential approach. Households were initially invited to participate via a push-to-web survey, followed by phone surveys and finally, face-to-face interviews for non-respondents.
To maximize data quality and minimize respondent burden, the lengthy questionnaire was divided into three parts, delivered to respondents in two-week intervals. This resulted in a complex protocol with an escalating number of communications and data management challenges during the collection phase, particularly between the sequential stages.
Two supplementary data collections were also conducted: one replicating the previous face-to-face only design and another using a full web-survey to bolster respondent numbers in specific areas.
This presentation will analyze the outcomes of the different survey designs, focusing on respondent characteristics and completion rates for each part of the questionnaire across all modes. Initial findings indicate a significantly higher response rate for the first part of the questionnaire compared to the traditional approach. However, due to attrition, the overall completion rate for the entire questionnaire shows a less pronounced difference between the old and new protocols.
We will discuss the critical role of the face-to-face phase and outline anticipated developments for the next survey iteration in 2028.