All time references are in CEST
Open Forms of Pretesting |
|
| Session Organisers | Dr Arne Bethmann (SHARE Germany and SHARE Berlin Institute) Ms Charlotte Hunsicker (SHARE Germany and SHARE Berlin Institute) Ms Dörte Naber (Universidad de Granada) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
The pretesting of survey questionnaires is crucial for ensuring high quality data in social research. In order to establish the validity of survey responses, it is essential to assess how respondents understand the questions and whether their understanding is consistent with the researcher's intended meaning. Over the years, Willis' (2005, 2015) original approach of cognitive pretesting, rooted in cognitive psychology, has been further developed and merged with other survey methodological and qualitative research approaches into new forms of open pretesting. These allow us to focus on respondents' understanding of survey questions by listening, to varying degrees, to what respondents actually think about the questions.
However, there is a trade-off between the richness of the data and the effectiveness of data collection and analysis. At one end of the spectrum is web-probing, which is highly efficient at collecting data on a large scale, but with a limited amount of information. At the other end are qualitative-interpretive approaches, such as grounded theory, which provide in-depth understanding at a high cost in time and resources. Classical cognitive pretesting, using e.g. standardised and emergent probes, falls between the two extremes, as do qualitative pretesting methods using more focused methodologies, such as problem-centred interviews.
In this session we bring together researchers who are developing and using different approaches across the spectrum of open pretesting. We are particularly interested in exploring the added complexity of different cultures and languages across countries. In addition, we will discuss how data collected using these methods can be used for further substantive research beyond the original purpose of pretesting survey questionnaires.
Keywords: open pretesting, questionnaire development, question meaning, cognitive pretesting
Ms Charlotte Hunsicker (SHARE Berlin Institute (SBI)) - Presenting Author
Dr Arne Bethmann (SHARE Berlin Institute (SBI))
Dr Christina Buschle (IU International University of Applied Sciences)
Dr Herwig Reiter (Geman Youth Institute (DJI))
Ms Theresa Fabel (SHARE Berlin Institute (SBI))
Dr Barbara Thumann (SHARE Berlin Institute (SBI))
The use of Qualitative Pretest Interviews (QPIs) in large-scale, multinational surveys like the Survey of Health, Ageing and Retirement in Europe (SHARE) provides valuable insights into respondents’ understanding of survey questions. Reflecting on the implementation of QPIs within SHARE, this presentation examines lessons learned across three key phases: conducting interviews, analysis, and implementing results.
1. Conducting Interviews: We reflect on the challenges and opportunities of conducting QPIs with an in-house team of interviewers who had specialized training and subject-matter familiarity. This phase highlights the importance of interviewer expertise in fostering intersubjective understanding with respondents, ensuring the collection of meaningful pretest data.
2. Analysis: Different qualitative approaches were explored to analyse the collected interviews, focusing on aligning these methods with the specific aims and contexts of the pretest. A key insight was balancing interpretive depth with the practical demands of conducting a large-scale, multinational survey.
3. Implementing Results: This section discusses how findings from QPIs were used to implement actionable changes in questionnaire design, highlighting the process of integrating qualitative feedback into the development of standardized survey instruments.
By reflecting on these three phases, the presentation provides a nuanced reflection on the role of QPIs in large-scale surveys, addressing both their potential and the challenges encountered.
Dr Raül Tormos (Centre d'Estudis d'Opinió - Generalitat de Catalunya) - Presenting Author
Ms Dörte Naber (Department of Methodology for Behavioral Sciences - University of Granada)
This paper investigates the use of online probing techniques, specifically web probing, as manipulation checks in survey experiments, with a focus on how experimental treatments may shift respondents' interpretations of survey questions. Using the example of political interest, we examine whether age-related primes can alter how individuals conceptualize their political engagement. We target the well-documented age gap in political interest, where younger people tend to report less interest than middle-aged individuals. Our study primes participants with questions on political topics tailored to various age groups (e.g., housing and job insecurity for the young, economic stability for older adults) before querying their general interest in politics. We hypothesize that respondents' interest levels will change based on how they interpret "politics" in light of the primes received. Subsequently, we use Web Probing to explore these interpretive shifts, assessing whether exposure to age-relevant issues can effectively bridge the age gap in reported political interest. Our findings are expected to demonstrate how priming certain topics can lead younger respondents to reinterpret political engagement through lenses relevant to their life experiences, thereby increasing their reported interest. This innovative approach underscores the diverse and underexplored opportunities cognitive probing offers, both by verifying that experimental manipulations operate as intended and by providing insights into how contextual information can shape responses in survey research. This study contributes to enhancing the accuracy of survey experiments and fosters a more nuanced understanding of age-related differences in political attitudes by refining the approach to measuring political interest.
Dr Herwig Reiter (German Youth Institute) - Presenting Author
Professor Christina Buschle (IU International University)
This paper introduces the rationale and practical steps of applying qualitative content analysis (QCA) in the frame of analysing data from qualitative pretest interviews (QPI). The QPI is a form of pretesting questionnaires and other standardised stimuli that has its methodological foundation in qualitative interview research (Buschle/Reiter/Bethmann 2022). By employing integrated communication strategies, the QPI involves interview partners as co-experts into a dialogic clarification of meaning, the identification of ambiguous or problematic formulations, and the exploration of possible improvements. The resulting material is similar to recordings and transcripts of thematically structured qualitative interviews. The purpose of the analysis of this material is the refinement and revision of the wording of questionnaire items and other standardised stimuli used in research. The analysis combines forms of group-based discursive analysis with coding and content analysis into procedures that gradually deepen and focus the interpretation on the most ambiguous aspects of the questionnaire, specific items or parts of standardised stimuli. After discussing the overall approach of analysing QPIs, this paper reflects the role of QCA in this process, and describes and illustrates how QCA is applied to interview sections indicating a need for revision.
Dr Vivian Meertens (Statistics Netherlands) - Presenting Author
In this presentation we reflect on pop up testing in comparison with traditional cognitive interviewing as a potential application for evaluating and testing methods to access the quality of data collection instruments. This presentation shows results of several pop up tests as an alternative and innovative method for pre-testing methods using by the Qlab of Statistics Netherlands. Several data collection products were tested like different versions of a landing page to log in on household surveys, different mobile designs of grid questions, versions of advance letters and statistical output products like figures and visualisations. Aspects like methods for recruitment, test locations and relevant practicalities will be discussed as well as reporting formats and data templates. The main question we reflect on is whether this innovative testing method can be a fruitful approach to test and evaluate data collection products?
Professor Kristen Miller (National Center for Health Statistics) - Presenting Author
Recently, the concept of respondent-driven survey practice has become a focus of attention. This presentation provides a framework specifically for respondent-driven question design. Using examples from several question design projects, the presentation lays out a definition for such a design along with methodological considerations.
Question and questionnaire design is often associated with a certain mystique, that to develop quality measures and instruments requires ‘artistic know-how’ and ‘common wisdom.’ This understanding solidified with Payne’s 1951 classic, The Art of Asking Questions, as he brought attention to the intricacy of question wording, arguing that what went into a question mattered. In the 1980s, the field of psychology brought attention to the cognitive aspects of survey measurement which, in part, opened the door for understanding question design as a scientific endeavor. By recognizing question response as a complex cognitive process, research into its four stages (comprehension, judgement, retrieval, response) could help to identify specific design features that would optimize respondents’ engagement and quality of their responses. Survey designers could then draw on a body of research about such topics as number of scale points, ‘don’t know’ options, order effects and recall error. Still, despite these efforts, an element of mystery remains, for in the Handbook of Survey Research (2010), Krosnick and Presser conclude that “the design of questions and questionnaires is an art as well as a science.”
This paper argues that respondent-driven design along with interpretive cognitive interviewing methodology, is another avenue for reducing ‘art,’ making question design a more objective process. Indeed, the ways in which respondents interpret and process survey questions is rooted in their personal experience that is tied to social location. Drawing upon respondents’ approach to questions can improve question design by ensuring that a question is similarly meaningful across contexts.
Dr Johanna Schütz (Kempten University of Applied Sciences) - Presenting Author
Ms Annika Hudelmayer (Kempten University of Applied Sciences)
Mr Kenneth zur Kammer (Kempten University of Applied Sciences)
Research on digitalisation is ubiquitous. In longitudinal research, maintaining the stability of questionnaires is a priority, which is challenging given the rapid pace of digitalisation.
In this context, the Qualitative Pretest Interview (QPI) method was used to adapt the German version of the “IT-Module” of the Survey of Health, Ageing and Retirement in Europe (SHARE). The module aims to record Internet access, use, and skills. Survey specialists were trained by qualitative interview specialists. In 2023 and 2024, 22 QPIs were conducted with people aged 50 to 87, engaging in a dialogue to clarify the meaning and understanding of the terminology used in the survey module.
The results showed, for example, that the participants had very different associations with the technical terms of digital technologies and found it difficult to match their personal digital activities to generic terms of a questionnaire. In addition, the results of the QPIs were analysed to determine whether they could be used for substantive research beyond the original purpose of pretesting the survey questionnaire. Using qualitative coding methods based on content analysis according to Kuckartz (2018), we identified themes such as the role of the family in the use of digital devices or stereotypes of ageing and digitalisation inherent to the interviewees and the interviewers.
The results contribute to the further development of quantitative instruments by involving the survey target group of older people as co-experts, leading to a more critical and validated interpretation of the results. Finally, we will discuss the challenges of integrating the results from qualitative pretests into a cross-country, multi-language survey.
Dr Celine Wuyts (Innovations for Poverty Action) - Presenting Author
Cognitive interviewing is rarely used in development policy research, particularly in low- and middle-income countries. This study adapts cognitive interviewing methods to explore how English-speaking respondents in Uganda, Nigeria, and the Philippines interpret survey items related to phone ownership, sharing, usage of digital financial services (DFS), and reasons for selecting or switching providers.
The socio-cultural contexts of these countries presented additional challenges, such as varying DFS literacy levels and the influence of hierarchical social structures on communication. In high power distance cultures, participants may struggle with thinking aloud or responding to cognitive probes, sometimes feeling disempowered or blaming themselves for difficulty verbalizing thoughts. To address this, we extended the rapport-building phase to help respondents feel more comfortable providing honest feedback.
This study demonstrates the value of cognitive interviewing, adapted to the context, in improving survey instruments in high power distance environments. Methodological lessons learned are discussed, along with recommendations for conducting cognitive interviews in similar contexts, emphasizing the importance of cultural sensitivity and respondent engagement in the pretesting process.
Ms Robyn Rapoport (SSRS) - Presenting Author
Ms Darby Steiger (SSRS)
Dr Paula Armendariz (SSRS)
Dr Nashay Lowe (SSRS)
Dr Sneha Mantri (Duke University)
Ms Esme Trahair (Duke University)
Qualitative methods featuring multi-phased designs are effective in deepening understanding of complex concepts and experiences and informing the development of survey instruments. Qualitative methods are flexible, allowing for mid-course adjustments to maximize learnings. This flexibility to tailor ‘fit-for-purpose’ designs is particularly valuable when conducting research with limited budgets and low incidence populations.
On behalf of the Michael J. Fox Foundation, SSRS conducted a multi-phase qualitative project to explore fears and uncertainties among people with Parkinson’s disease (PwP) regarding Parkinson’s disease and its progression. Our original project plan assumed the first phase would consist of 20 in-depth interviews (IDIs) to inform the development of a survey instrument to be cognitively tested in the second phase. After completing 11 interviews, the research team concurred data saturation was reached for the goals of this study.
Rather than completing the nine remaining interviews with the same protocol, the team determined survey development efforts would be better served by revising the discussion guide, to allow for a more focused, second phase of IDIs. Informed by findings from the initial interviews and prior work, we developed a list of fears and uncertainties that might resonate for PwP. For the remaining nine IDIs, participants were shown this list of items and asked which resonated most for them. This guided discussion enabled the research team to refine the lexicon of fear/uncertainty by asking participants to reflect on specific terminology and classifications of fears. These findings informed the development of a survey instrument that was cognitively pre-tested. Cognitive pretests were aimed at finalizing the survey content and gathering usability feedback on the programmed instrument.
In our presentation, we reflect on this case study and identify considerations for broader applications of using a flexible, iterative approach to qualitative inquiry.