Online probing: Cognitive interviewing techniques in online surveys and online pretesting 2 |
|
| Chair | Dr Katharina Meitinger (GESIS Leibniz Institute for the Social Sciences ) |
| Coordinator 1 | Dr Dorothée Behr (GESIS Leibniz Institute for the Social Sciences) |
| Coordinator 2 | Dr Lars Kaczmirek (GESIS Leibniz Institute for the Social Sciences) |
Cognitive interviewing is captured as the golden standard to improve and evaluate questionnaires. However, this method has also its downsides such as small sample sizes, high costs and the risk of interviewer effects. Online probing as a new development in survey methodology was implemented to address these problems (Behr et al., 2012, 2014). Online probing is normally realized using a web questionnaire in which respondents are asked cognitive probes after they have answered the closed item. Recent research indicates that online probing is an effective method detecting problematic questions (Behr et al., 2012; 2014; Meitinger & Behr, 2016). However, until now most research was based on attitudinal-questions (civil disobedience, gender ideology etc.), raising the question if online probing is also suitable for factual-questions and in detail, which probing technique may be the best for factual-questions in order to accomplish answers with the most analytical potential.
The present contribution addresses these questions by implementing online probing for the Labour Force Survey (LFS). Moreover, this contribution is to our knowledge the first, comparing the effectiveness of different online probing techniques (confidence rating, comprehension and category-selection probes) for factual-questions.
The sample was recruited from the LFS CATI-sample. Wave 1 of data collection is completed, wave 2 is actually realized. In sum, 145 respondents completed the web questionnaire until now. In wave 2 another 150 completed questionnaires are expected.
Results show that depending on the question-type, different probing techniques may be preferred. Asking the ISCO question confidence rating yield slightly more characters compared to category-selection probes (46 vs. 44 characters). Contradictory, category-selection probes compared to confidence rating and comprehension probes (48 vs. 41 vs. 33 characters) may be better suited for the question of respondents’ actual labour status (WSTATOR). Comparing confidence rating with a non-specific probing question, results of the NACE-question show that confidence rating fits the question best (45 vs. 34 characters). To test for significance, presented results will be expanded by data of wave 2. Furthermore, results will be enriched by an analysis of the quality of the given answers.
In sum, results indicate that online probing is a valuable tool to improve factual-questions. However, online probing does not provide the possibility to ask proper follow-up probes which limits the results by far. Thus, we believe in line with others (Behr et al., 2014) that traditional methods such as face-to-face cognitive interviews may keep the golden standard when it comes to the improvement and the evaluation of factual-questions. However, online probing may be used as a possible solution to get results faster, at a reduced rate and from a broader population. Hence, we conclude that online probing can be seen as a well-established tool for an initial or follow up diagnosis of questions, yet, the in-depth analysis should be completed by face-to-face cognitive interviews.
Embedding cognitive probes in online surveys is a relatively low-cost method that allows question and questionnaire designers and evaluators to obtain a large amount of cognitive information in an efficient manner. This is particularly the case in large-scale national and cross-national surveys, where the geographic limitations of typical face-to-face cognitive interviewing samples are more pronounced. The majority of methodological work on online probing has focused on the use of open-ended probes, which are designed to collect qualitative information that is comparable to that which is obtained from cognitive interviewing (Behr et al 2014, Fowler et al 2016, Meitinger and Behr 2016).
Besides open-ended probes; however, more structured, close-ended probes should also be considered when designing online probing studies. Unlike open-ended ones, close-ended probes do not produce primary qualitative information, but rather rely on and expand the results of previous qualitative studies (such as a cognitive interviewing project). In doing so, they present a more streamlined and less burdensome question to survey respondents, while allowing for the results of qualitative question evaluation studies to be extrapolated to a full survey population (Maitland et al 2013, Scanlon 2016).
Unfortunately, close-ended online probes have been explored in much less methodological detail than open-ended ones. This presentation will attempt to close this gap by presenting some early methodological results and suggesting new lines of inquiry using data from multiple rounds of the National Center for Health Statistics’ (NCHS) Research and Development Survey (RANDS), a web survey conducted on a probability panel of American adults. Outcome measures such as probe response rates, item non-response and breakoff rates, and category selection behaviors will be used to explore issues such as probe placement, repetition and answer category design. Building on the results of these early methodological studies, a few best practices in the use of these embedded, close-ended probes will be presented, illustrating how their use can fit into wider national and cross-national question evaluation studies.
In standard cognitive pretesting, interviewers ask respondents to ‘think aloud’ and to answer verbal probes. The purpose is to understand how respondents comprehend and respond to survey questions, with the aim of fixing problems and enhancing data quality. Typically, small samples of five to fifteen participants are used and questions revised (or not) in light of what is found. Although cognitive testing is well-established, quite serious questions remain as to its efficacy, which primarily have to do with the small Ns involved. Firstly, it is possible that not all important problems with a survey item will be uncovered by a small number of interviews. Recent research suggests that indeed many problems may routinely be missed when small samples are employed (Blairand Conrad 2011). Secondly, it is generally not known how significant or widespread such problems might be for the study population as a whole. Thirdly, and following on from the second, it is not known how changes made to questions affect data quality. Web surveys have begun to be used to carry out cognitive testing online, using open ended probes, in order to solve some of these issues (Behr et al), but in no case to our knowledge has a probability-based design been used. In this study, we run a randomised experiment on the NatCen online probability-based panel survey. We use closed-ended probes to evaluate revisions made to survey items after they have undergone standard cognitive pretesting. Analysis suggests modest success for standard cognitive testing methods, at least in the cases we examine.