All time references are in CEST
Overcoming challenges in mobile questionnaire design 1 |
|
| Session Organiser | Ms Joanna d'Ardenne (NatCen Social Research) |
| Time | Wednesday 19 July, 11:00 - 12:30 |
| Room | U6-01c |
Research commissioners are increasingly interested in offering an online alternative to surveys that have historically relied on face-to-face data collection. Transitioning a CAPI survey to an online survey can pose practical challenges when it comes to questionnaire design. One challenge is the principle of mobile-first design. CAPI questionnaires will have been designed to be completed by a trained interviewer using a large screen device. Web questionnaires, in contrast, must be accessible to novice users, including those who are participating on small screen devices such as smartphones. Poor mobile design can increase respondent burden and frustration. This increases the risk of missing data and break-off.
Although it is straightforward to render simple questions on mobile screens, some CAPI questions require redesign work to make them smartphone appropriate. In this session we invite survey practitioners to showcase any work they have undertaken to develop mobile versions of more challenging CAPI questions. Examples could include, but are not limited to, the following:
- Questions that include lengthy text or complex instructions
- Household enumeration questions
- Grid based or looped questions
- Questions that make use of hidden codes
- Event History Calendars
We invite practitioners to present findings on their redesign process, which designs were favorable, which were unfavorable, and the methods used for pre-testing their refined mobile-first questions.
Keywords: Questionnaire, mobile, pre-testing, usability, web, online, mode, mode transition
Dr Lin Wang (U.S. Census Bureau) - Presenting Author
Dr Shelley Feuer (U.S. Census Bureau)
Dr Alda Rivas (U.S. Census Bureau)
Dr Heather Ridolfo (U.S. Energy Information Administration)
Dr Anthony Schulzetenberg (LexisNexis Corporation)
Diary surveys collect detailed information from respondents over a period of time that could shed light on patterns of human behaviors or societal activities (e.g., Consumer Expenditure Diary Survey, American Time Use Survey). Because of the extended data collection period and large amount of data to be collected, data loss and data accuracy (non-response error and measurement error) are two particular concerns to survey designers. An optimally designed survey instrument could provide an effective remedy to this challenge. Smartphones provide an opportunity to reduce respondent burden and improve data quality in diary surveys. In this paper, we propose an approach of human-centered design to developing a mobile diary survey that help the respondent complete diary surveys effectively, efficiently, and with satisfaction. Within the context of a large-scale national diary survey study, three aspects in mobile diary survey development will be highlighted: workflow design, usability evaluation, and instrument use training. For workflow design, we discuss how to gather empirical data through respondent research to inform instrument design. A study of sequential card sorting will be presented as an example. For usability evaluation, we describe the lifecycle of iterative usability testing including participants sample design, test scenarios, and performance measurement. For instrument use training, we focus on training curriculum design and training evaluation and discuss the pros and cons of in-person and video-based training. Finally, we conclude with lessons learned about designing and developing a mobile diary survey and how these lessons can be applied to other mobile diary surveys.
Dr Arieke Rijken (Netherlands Interdisciplinary Demographic Institute)
Dr Siyang Kong (Netherlands Interdisciplinary Demographic Institute)
Dr Wojciech Jablonski (Netherlands Interdisciplinary Demographic Institute) - Presenting Author
The Generation and Gender Survey (GGS) provides longitudinal and cross-national data about population and family dynamics. In 2020, GGS started a new round of data collection (GGS-II) with a new sample in each participating country. Whereas CAPI (computer- assisted personal interviewing) was the only mode of data collection in GGS-I, GGS-II is mixed-mode. Web interviewing has become the dominant mode of data collection. Several countries have conducted Wave 1 of GGS-II as a web-only survey, either as a push-to web survey using an offline contact mode or using an online contact mode. Other countries used web interviewing in combination with CAPI, PAPI (paper-and-pencil interviewing), or CATI (computer-assisted telephone interviewing) as fallback modes in order to decrease non-response.
In this paper we will focus on one particular challenge of web surveys: the fact that nowadays web surveys are mixed-device surveys. On the one hand, the fact that respondents can use their preferred device to participate has advantages, on the other hand the use of mobile devices with small screens may pose challenges for completing a long and demanding questionnaire like that of the GGS. We will combine paradata on the used device with survey data and give an overview of device use (smartphone; tablet; laptop/desktop) by country. Furthermore, we will investigate self-selection of respondents into using an particular device and the impact of device on interview duration and data quality (e.g. break offs, item-nonresponse).
Ms Irina Bauer (GESIS – Leibniz Institute for the Social Sciences) - Presenting Author
Event History Calendars were found to improve the quality of retrospective biographical data in terms of reported episodes and completeness of data for interviewer-administered surveys. But with an increasing number of surveys being conducted in self-administered modes, tools for collecting retrospective data need to be adapted to enable respondents to utilize them without interviewer assistance. Thus, two major changes in the collection of biographical data result from this development: First, the shift from interviewer-administered to self-administered surveys, and second, the need to adapt traditional tools for collecting retrospective data to be suitable for mobile devices as well. Understanding how this affects the quality of biographical data is crucial. Consequently, the present study aims at answering the following research question: 1) Does switching from an EHC to an alternative tool to collect biographical data affect data quality? 2) How does the absence of an interviewer influence the collection of biographical data?
To investigate these research questions, I draw on two waves of the German family panel pairfam, which underwent a switch from an interviewer-administered EHC (field period: October 2018 to May 2019) to an alternate version, that was either administered by an interviewer (N_CAPI=926) or in a self-administered web survey (N_CAWI=4,066; October 2020 to February 2021). Respondents were randomly assigned to the different modes allowing potential effects resulting from the mode switch to be disentangled from those due to the change in the tool used to collect occupational data from the respondents. To assess RQ1, I compare data quality for respondents who were interviewed via CAPI in both waves. To assess RQ2, I compare both mode groups of the latter wave. To capture data quality, I examine the number of reported episodes as well as the completeness of the episodes.
Ms Jo d'Ardenne (NatCen Social Research) - Presenting Author
Mr Curtis Jessop (NatCen Social Research)
The Skills and Employment Survey (SES) 2023 will be the eighth in a series of surveys, stretching back over 35 years, with workers in Britain about their working conditions. The survey aims to capture changes in job quality i.e. employee involvement, work intensity, job security, training and whether the job is a good match for employee skills. All prior SES waves have been conducted using face-to-face interviews; the last SES having been conducted in 2017. Interview length is approximately 60 minutes.
In 2023, in addition to having face-to-face data collection, the SES is going to be trialed on an online panel for the first time. This is because, although continuity is critical for the SES, there is a desire to explore alternative data collection methods. If it is possible to collect robust data online more regular data collection may become possible.
In this presentation we will showcase the process via which the questionnaire was modified to make it appropriate for web-based administration. We will demonstrate our five-step approach to converting a questionnaire to another mode and how we applied it to this specific instrument. We will focus on ‘mobile pain points’ i.e. the parts of the original questionnaire that were most challenging to convert to a mobile friendly design, and our proposed solutions to the problems we encountered.