Recent Developments in Question Testing 1 |
|
| Session Organisers |
Dr Cornelia Neuert (GESIS - Leibniz Institute for the Social Sciences) Dr Timo Lenzner (GESIS - Leibniz Institute for the Social Sciences) |
| Time | Thursday 18th July, 09:00 - 10:30 |
| Room | D17 |
It is universally acknowledged that testing survey questions prior to administering them to respondents is a vital part of a survey as pretesting reduces potential measurement error and helps to improve the quality of the data collected. For question testing, survey methodologists have a broad set of methods at their disposal (cognitive interviews, behavior coding, response latency measurement, vignettes, expert reviews). Recently, innovative techniques and new data sources are added to the survey researcher’s toolbox, such as eye tracking, web probing, mouse movements or crowdsourcing.
So far, few methodological studies have addressed the effectiveness of these newer methods in improving questionnaires and how they compare to traditional pretesting methods.
This session invites papers that…
(1) explore innovative uses of new methods or techniques for question testing;
(2) highlight the relative effectiveness of different pretesting methods
(3) demonstrate how new and existing techniques might best be used in combination (best-practice examples) to offer additional insights.
We also invite presentations discussing (new) question testing methods in a cross-cultural context.
Keywords: Question testing, Web Probing, Cognitive interviewing, Eye Tracking, Pretesting Methods
Ms Margaret Roller (Roller Research) - Presenting Author
Ms Karen Kellard (Social Research Centre)
Although survey researchers have debated quality in their designs for decades, a quality approach to the cognitive interviewing method is new. Verification strategies, the role of reliability and validity, and attention to “qualitative rigor” are well-documented in the literature. In 2015, a new approach was introduced that incorporates research principles at each phase of the qualitative research process without stifling the goals of inquiry, creative methods, and interpretations utilized by qualitative researchers. This approach is the Total Quality Framework (TQF).
The TQF offers a conceptual foundation from which qualitative researchers are able to think about the quality of their cognitive interviewing designs. The TQF is rooted in the belief that cognitive interviewing and all qualitative research methods must be: credible, analyzable, transparent, and useful. These core objectives represent the four interconnected components of the TQF: Credibility (the completeness and accuracy of the data); Analyzability (the processing and verification of the data); Transparency (the completeness [“thickness”] of disclosure in the final research document, and transferability); and Usefulness (the ability to do something of value with the outcomes).
Utilizing best practices within each component provides qualitative researchers conducting cognitive interviews a way to (a) give explicit attention to quality issues, (b) critically examine the possible sources of variability and bias, (c) incorporate features into their designs that try to mitigate these effects, and (d) take the implications of these effects into consideration during analysis.
This presentation defines the TQF components, provides examples of best practices in the cognitive interviewing method, and discusses the interrelationship between the components and the ultimate value of this approach.
Ms Karen Kellard (The Social Research Centre, Australian National University) - Presenting Author
This paper will contribute to discussions on the relative effectiveness of different qualitative pre-testing approaches - focus groups and individual in-depth interviews (IDIs) for the cognitive testing of survey questions. Whilst the use of IDIs for cognitive interviews is a well-recognised and used technique, the researchers have observed that for some individuals (such as young people or others who may be vulnerable) cognitive testing can be somewhat daunting, even with very skilled interviewers (potentially making it challenging to uncover measurement errors). Such individuals appear to be more comfortable and relaxed in a focus group setting with other similar people. Nevertheless, cognitive testing in focus groups has both known and unknown limitations, not least conforming behaviours and a reluctance to disagree with group members, thus hiding areas of measurement error.
Drawing on findings from a recent small experiment (testing the same set of questions in groups and individual interviews) conducted by qualitative researchers at the Social Research Centre (Australian National University), we will review the relative strengths and weaknesses of each approach in identifying measurement errors through the assessment of comprehension, judgement, retrieval and response, and provide an overview of when the two different approaches may work best.
Dr Christina Buschle (FernUniversität in Hagen)
Dr Arne Bethmann (Max-Planck-Institute for Social Law and Social Policy) - Presenting Author
Dr Herwig Reiter (German Youth Institute)
Published on Zenodo with DOI: https://doi.org/10.5281/zenodo.3342094
Cognitive interviews have become one of the most important pretest procedures in the development and evaluation of questionnaires. Different techniques such as thinking aloud or different probing approaches examine the comprehensibility of questions and statements and uncover the reasons for difficulties of respondents in answering questionnaires. What is missing in cognitive interviewing is the methodological framing of these techniques. Against this background our contribution proposes such a methodological framing on the basis of similarities of cognitive interviewing procedures with qualitative interviewing strategies. Both involve direct research interactions that can be regarded as communication processes between interviewer and respondent based on intersubjective understanding (Fremdverstehen). We introduce methodically integrated communication strategies of two established qualitative interviewing techniques - the problem-centred interview and the discursive interview - and discuss their potential for the development of a pretest interview approach, which includes the techniques of the cognitive interview and accounts for the essentially social character of processes of clarifying comprehension. We suggest the term Qualitative Pretest Interview (QPI) in order to avoid a narrow understanding of pretest procedures as focused on the problem of ambiguous cognitions. Finally, we discuss the potential practical benefits of this approach of interviewer-respondent interaction for standardized survey research and provide examples from QPIs that were conducted during actual questionnaire development.