All time references are in CEST
Methods for questionnaire development: SQP, Cognitive Interviewing, and others 1 |
|
| Session Organisers | Dr Cornelia Neuert (GESIS - Leibniz Institute for the Social Sciences) Dr Lydia Repke (GESIS - Leibniz Institute for the Social Sciences) |
| Time | Thursday 20 July, 09:00 - 10:30 |
| Room | U6-28 |
There are various methods and procedures available for evaluating survey items during questionnaire development. They all share the goal to help researchers to identify questions that are flawed, difficult to comprehend, and might lead to measurement error. However, the available methods differ in several aspects, such as timing, the outcome, the amount of data collection needed, and whether respondents, experts, or computer systems are involved.
In practice, considerations like the available time and budget, the target population, and the mode of data collection affect the choice.
As there is still no clear consensus about best practices for question evaluation, this session aims to discuss current questionnaire evaluation practices using different methods and tools. We invite papers that use different types of evaluation methods, such as expert reviews and cognitive interviewing, or the Survey Quality Predictor (SQP; https://sqp.gesis.org).
We invite papers that
(1) exemplify which evaluating method might best be used during questionnaire development;
(2) assess the different evaluation methods with respect to their advantages and disadvantages;
(3) provide information on how methods can best be used in combination.
Keywords: questionnaire development, question evaluation methods, pretesting, cognitive interviewing, SQP, expert reviews, web probing, behavior coding
Ms Charlotte Hunsicker (MEA-SHARE and SHARE Berlin Institute) - Presenting Author
Dr Arne Bethmann (MEA-SHARE and SHARE Berlin Institute)
Professor Christina Buschle (IU International University of Applied Sciences)
Dr Herwig Reiter (German Youth Institute (DJI))
The pretesting of draft questionnaires is usually outsourced to survey agencies and supervised by research institutes. The paper introduces the rationale behind the alternative of establishing an in-house team of embedded interviewers for doing Qualitative Pretest Interviews (QPI) and illustrates the first steps of its establishment in the frame of the Survey of Health, Aging and Retirement in Europe (SHARE). SHARE is a multi-disciplinary and cross-national panel research infrastructure for studying the effects of health, social, economic and environmental policies over the life-course in 28 European countries and Israel.
The QPI is a novel method of pretest interviewing for improving the quality of standardised research. By advocating a comprehensive consideration of qualitative-interpretive methodology, the QPI transfers the idea of intersubjective understanding in everyday communication to the situation of pretest interviewing. Following the requirements of qualitative research the QPI relies heavily on the familiarity of researchers and interviewers with all aspects of the research topic as well as the purpose of single items and item batteries. Therefore, the establishment and training of an in-house team for pretest-interviewing combines the advantages of substantive expertise with the possibility of immediate and stepwise revision of standardised instruments. This alternative can improve, accelerate and focus the process of pretesting. The paper concludes with a discussion of advantages an limits of this alternative.
Dr Marina Aleksandrova (NRU HSE) - Presenting Author
A variety of methods aimed at assessing and improving the quality of questionnaires continue to be developed by sociologists. One of the promising areas for improving questionnaires is to work on reducing the frequency of item nonresponse in surveys. While most studies are focused on dealing with item nonresponse error (INE) when survey data has already been collected, almost no attention was given to investigate how this type of error can be predicted prior to data collection process. Different characteristics, which can influence respondent's willingness to answer questions in surveys, were studied in scientific literature, and among them, two approaches can be distinguished: research of characteristics related to respondents, and research of characteristics related to the questionnaire. These two approaches were never investigated together, despite existing assumptions that respondents with different sociodemographic characteristics respond differently to different wordings of survey questions. In this paper, we investigate how gender, age and financial situation as sociodemographic characteristics of respondents, and characteristics of questionnaires (question wording) simultaneously affect the occurrence of INE in the sociological data. We achieve the research goal using natural language processing (NLP) and machine learning methods. We develop a methodology to organize and preprocess the data with survey questions, and sociodemographic characteristics of respondents. Three machine learning algorithms were compared (Binary Logistic Regression, Gradient Boosting, Random Forest) to choose best models based on the quality metrics and interpret results. The database of the study is ESS (European Social Survey), which were conducted in Great Britain in English language, which was used to train models of prediction the INE.
Results of the conducted research include description of specifics of response behavior of respondents depending on question wording and sociodemographic characteristics, and the model of prediction of probability of item nonresponse occurrence depending on the formulation of question.