Potential and Challenges of Cognitive Interviewing and Online Probing in a Cross-National Context |
|
| Convenor | Mr Andrew Johnson (Ipsos MORI ) |
| Coordinator 1 | Ms Katarina Meitinger (GESIS) |
Cognitive interviewing is most effective when respondents verbalize their thought process in response to the survey questions and interviewer probes. This process may be unfamiliar and difficult for some respondents, particularly in contexts outside of the United States and Europe where there is little survey tradition. Little guidance exists in the literature on how to apply cognitive interviewing in such contexts. This paper examines the effect of providing respondents with examples and practice think-alouds before the main cognitive interview in an experiment carried out in rural Nepal where experience with surveys and cognitive interviewing is rare.
Cognitive interviewing allows us to explore how respondents interpret survey questions and assess whether they are understood as intended. It is increasingly used to pre-test questions developed for cross-national surveys to ensure equivalence of meaning in different languages and cultures. In this context, achieving standardisation in the way that interviews are conducted and the data are organised and analysed is an important but challenging task. This paper will review the approaches that have been used to collect, organise and analyse data from cognitively testing questions for several European surveys and suggest how improvements could be made.
The presentation addresses potential challenges and drawbacks of conducting cognitive interviews in a multi-national, multi-cultural contexts by identifying both issues of comparability and conceptual equivalence and the challenges that researchers face when trying to apply a standardised uniform approach, including a systematic comparative analysis across diverse settings.
The presentation is based on a cross-country/ cross-group database of 280 cognitive interviews collected in the framework of an extensive cognitive pre-test study, conducted in 2014 in eight EU Member States to inform the development of the second European Union Minorities and Discrimination Survey, which the European Union
Questions asking respondents to report self-rated health and subjective life expectancy are commonly included in cross-national surveys. While both items have been shown to be strong predictors of actual mortality and health, each encompasses distinct and complex tasks of comprehension, retrieval, judgment, and mapping. Thus, investigating cognitive processes by which respondents arrive at their answers may be key to understanding the utility of these questions in general. We present results from cognitive probing items implemented in a cross-national Internet survey, fielded in five countries and three languages, to examine whether cultural context frames and influences response.
Cross-national studies investigating elements of national identity frequently use the item “How proud are you of being German/American/British?“ as a substitute to measure either patriotism or nationalism. However, it is questionable that this item is a valid measure for these complex constructs because respondents in different countries could associate various constructs with it.
The presentation will scrutinize the appropriateness of the general pride item as a measure of nationalism and/or patriotism. It will combine quantitative results from the ISSP 2013 with online probing results from a web survey conducted with 2,685 respondents in five countries.