ESRA 2025 Preliminary Program
All time references are in CEST
Open Forms of Pretesting 2 |
| Session Organisers |
Dr Arne Bethmann (SHARE Germany and SHARE Berlin Institute)
Ms Charlotte Hunsicker (SHARE Germany and SHARE Berlin Institute)
Ms Dörte Naber (Universidad de Granada)
|
| Time | Thursday 17 July, 14:00 - 15:00 |
| Room |
Ruppert 111 |
The pretesting of survey questionnaires is crucial for ensuring high quality data in social research. In order to establish the validity of survey responses, it is essential to assess how respondents understand the questions and whether their understanding is consistent with the researcher's intended meaning. Over the years, Willis' (2005, 2015) original approach of cognitive pretesting, rooted in cognitive psychology, has been further developed and merged with other survey methodological and qualitative research approaches into new forms of open pretesting. These allow us to focus on respondents' understanding of survey questions by listening, to varying degrees, to what respondents actually think about the questions.
However, there is a trade-off between the richness of the data and the effectiveness of data collection and analysis. At one end of the spectrum is web-probing, which is highly efficient at collecting data on a large scale, but with a limited amount of information. At the other end are qualitative-interpretive approaches, such as grounded theory, which provide in-depth understanding at a high cost in time and resources. Classical cognitive pretesting, using e.g. standardised and emergent probes, falls between the two extremes, as do qualitative pretesting methods using more focused methodologies, such as problem-centred interviews.
In this session we bring together researchers who are developing and using different approaches across the spectrum of open pretesting. We are particularly interested in exploring the added complexity of different cultures and languages across countries. In addition, we will discuss how data collected using these methods can be used for further substantive research beyond the original purpose of pretesting survey questionnaires.
Keywords: open pretesting, questionnaire development, question meaning, cognitive pretesting
Papers
Using Online Probing Techniques as Manipulation Checks: Exploring Age-Specific Priming Effects on Political Interest in Survey Experiments
Dr Raül Tormos (Centre d'Estudis d'Opinió - Generalitat de Catalunya) - Presenting Author
Ms Dörte Naber (Department of Methodology for Behavioral Sciences - University of Granada)
This paper investigates the use of online probing techniques, specifically web probing, as manipulation checks in survey experiments, with a focus on how experimental treatments may shift respondents' interpretations of survey questions. Using the example of political interest, we examine whether age-related primes can alter how individuals conceptualize their political engagement. We target the well-documented age gap in political interest, where younger people tend to report less interest than middle-aged individuals. Our study primes participants with questions on political topics tailored to various age groups (e.g., housing and job insecurity for the young, economic stability for older adults) before querying their general interest in politics. We hypothesize that respondents' interest levels will change based on how they interpret "politics" in light of the primes received. Subsequently, we use Web Probing to explore these interpretive shifts, assessing whether exposure to age-relevant issues can effectively bridge the age gap in reported political interest. Our findings are expected to demonstrate how priming certain topics can lead younger respondents to reinterpret political engagement through lenses relevant to their life experiences, thereby increasing their reported interest. This innovative approach underscores the diverse and underexplored opportunities cognitive probing offers, both by verifying that experimental manipulations operate as intended and by providing insights into how contextual information can shape responses in survey research. This study contributes to enhancing the accuracy of survey experiments and fosters a more nuanced understanding of age-related differences in political attitudes by refining the approach to measuring political interest.
Analysing Qualitative Pretest Interviews using Qualitative Content Analysis
Dr Herwig Reiter (German Youth Institute) - Presenting Author
Professor Christina Buschle (IU International University)
This paper introduces the rationale and practical steps of applying qualitative content analysis (QCA) in the frame of analysing data from qualitative pretest interviews (QPI). The QPI is a form of pretesting questionnaires and other standardised stimuli that has its methodological foundation in qualitative interview research (Buschle/Reiter/Bethmann 2022). By employing integrated communication strategies, the QPI involves interview partners as co-experts into a dialogic clarification of meaning, the identification of ambiguous or problematic formulations, and the exploration of possible improvements. The resulting material is similar to recordings and transcripts of thematically structured qualitative interviews. The purpose of the analysis of this material is the refinement and revision of the wording of questionnaire items and other standardised stimuli used in research. The analysis combines forms of group-based discursive analysis with coding and content analysis into procedures that gradually deepen and focus the interpretation on the most ambiguous aspects of the questionnaire, specific items or parts of standardised stimuli. After discussing the overall approach of analysing QPIs, this paper reflects the role of QCA in this process, and describes and illustrates how QCA is applied to interview sections indicating a need for revision.
Potential of Qualitative Pretest Interviews for In-depth Research Insights into Older Adults' Internet Use
Dr Johanna Schütz (Kempten University of Applied Sciences) - Presenting Author
Ms Annika Hudelmayer (Kempten University of Applied Sciences)
Mr Kenneth zur Kammer (Kempten University of Applied Sciences)
Research on digitalisation is ubiquitous. In longitudinal research, maintaining the stability of questionnaires is a priority, which is challenging given the rapid pace of digitalisation.
In this context, the Qualitative Pretest Interview (QPI) method was used to adapt the German version of the “IT-Module” of the Survey of Health, Ageing and Retirement in Europe (SHARE). The module aims to record Internet access, use, and skills. Survey specialists were trained by qualitative interview specialists. In 2023 and 2024, 22 QPIs were conducted with people aged 50 to 87, engaging in a dialogue to clarify the meaning and understanding of the terminology used in the survey module.
The results showed, for example, that the participants had very different associations with the technical terms of digital technologies and found it difficult to match their personal digital activities to generic terms of a questionnaire. In addition, the results of the QPIs were analysed to determine whether they could be used for substantive research beyond the original purpose of pretesting the survey questionnaire. Using qualitative coding methods based on content analysis according to Kuckartz (2018), we identified themes such as the role of the family in the use of digital devices or stereotypes of ageing and digitalisation inherent to the interviewees and the interviewers.
The results contribute to the further development of quantitative instruments by involving the survey target group of older people as co-experts, leading to a more critical and validated interpretation of the results. Finally, we will discuss the challenges of integrating the results from qualitative pretests into a cross-country, multi-language survey.
Refining Insights to Better Understand Fears and Uncertainties for People with Parkinson’s Disease: Leveraging Iterative Qualitative Research to Develop and Pretest a Survey Instrument
Ms Robyn Rapoport (SSRS) - Presenting Author
Ms Darby Steiger (SSRS)
Dr Paula Armendariz (SSRS)
Dr Nashay Lowe (SSRS)
Dr Sneha Mantri (Duke University)
Ms Esme Trahair (Duke University)
Qualitative methods featuring multi-phased designs are effective in deepening understanding of complex concepts and experiences and informing the development of survey instruments. Qualitative methods are flexible, allowing for mid-course adjustments to maximize learnings. This flexibility to tailor ‘fit-for-purpose’ designs is particularly valuable when conducting research with limited budgets and low incidence populations.
On behalf of the Michael J. Fox Foundation, SSRS conducted a multi-phase qualitative project to explore fears and uncertainties among people with Parkinson’s disease (PwP) regarding Parkinson’s disease and its progression. Our original project plan assumed the first phase would consist of 20 in-depth interviews (IDIs) to inform the development of a survey instrument to be cognitively tested in the second phase. After completing 11 interviews, the research team concurred data saturation was reached for the goals of this study.
Rather than completing the nine remaining interviews with the same protocol, the team determined survey development efforts would be better served by revising the discussion guide, to allow for a more focused, second phase of IDIs. Informed by findings from the initial interviews and prior work, we developed a list of fears and uncertainties that might resonate for PwP. For the remaining nine IDIs, participants were shown this list of items and asked which resonated most for them. This guided discussion enabled the research team to refine the lexicon of fear/uncertainty by asking participants to reflect on specific terminology and classifications of fears. These findings informed the development of a survey instrument that was cognitively pre-tested. Cognitive pretests were aimed at finalizing the survey content and gathering usability feedback on the programmed instrument.
In our presentation, we reflect on this case study and identify considerations for broader applications of using a flexible, iterative approach to qualitative inquiry.
