ESRA 2025 Preliminary Program
All time references are in CEST
Web Probing |
| Session Organisers |
Dr Katharina Meitinger (Utrecht University)
Dr Dorothée Behr (GESIS)
Dr Paul Scanlon (CDC)
|
| Time | Wednesday 16 July, 09:00 - 10:30 |
| Room |
Ruppert D - 0.24 |
Web probing – that is, the implementation of probing techniques from cognitive interviewing in web survey with the goal to assess the validity of survey items – is a valuable addition to the toolbox of (cross-cultural) survey methodologists (Behr et al. 2017). Thanks to the implementation in web surveys, web probing can have large sample sizes which allow for an analysis of response patterns in subpopulations and a prevalence assessment of question problems and themes. The method has already been used to assess measurement instruments for a large variety of topics and methodological research has already addressed several aspects of web probing, such as optimal design or nonresponse detection.
However, several research gaps remain, such as:
- Optimal web probing design
- Prevention strategies to reduce nonresponse and mismatches
- Samples for web probing and implications for data quality
- Probe types and their potential to generate insights into the cognitive response process
- Level of granularity of web probing studies (e.g., regional analysis)
- Performance of web probing in non-European/American contexts
- Data collection with voice and video recordings
- The use of AI for the translation of probe responses
- The use of AI for the coding and analysis of probe responses
For this session, we invite (1) presentations with a substantive application of web probing and (2) presentations that address some of the methodological challenges and considerations of web probing.
Keywords: web probing, pretesting, cross-cultural, AI, optimal design
Papers
Enhancing Cross-national Comparability of the Measurement of Solidarity in the European Values Study
Miss Margherita Pellegrino (University of Bergamo) - Presenting Author
Professor Vera Lomazzi (University of Bergamo)
Recent studies assessing measurement equivalence of the 9-items battery used in the European Values Study (EVS) to measure solidarity have highlighted its weak cross-national comparability. The measurement consists of a question asking respondents to express their concern for the living conditions of a series of social groups. Lack of invariance is mainly due to the items related to “Europeans” and “Immigrants”. Also, the term “concern” used in the question wording resulted challenging.
To uncover these biases and contribute improving the clarity of questions in 2026 EVS questionnaire, we conducted web probing in 4 countries (Italy, Czechia, Portugal, Hungary) via online panel (national quota samples by sex age group, educational level, age group of population 18-64 years old; 600 cases (150 cases/country) and inserted specific probes to explore the variation in respondents’ interpretation of the terms “Europeans,” “immigrants,” and “concern”. Additionally, we used category-selection probe to understand respondents’ reason for expressing a certain level of concern toward Europeans. The data was translated and categorized using thematic coding across languages. For each response, multiple categories were identified, demonstrating the diverse interpretations of the same words within and especially across countries, despite the accurate translation of the EVS questionnaire. For instance, in Italy and Hungary, the word "concern" was primarily interpreted as "cherish" or "care," whereas in Portugal, it was more commonly associated with being "worried." Similarly, the term "immigrants" was understood in multiple ways. In Hungary and Portugal, it was often perceived as referring to generic migrants, whereas in Czechia and Italy, respondents specified categories. This study demonstrates the value of web probing as a tool for identifying and addressing cultural biases in international surveys. The insights gained provide a basis for refining survey instrument.
Differential Data Quality Across Population and Methods-Based Subgroups in Open-Ended Web Probes
Dr Paul Scanlon (National Center for Health Statistics) - Presenting Author
One of the largest use cases of web panels in survey development and evaluation is embedding cognitive probes (often referred to as “web probes”) in a survey questionnaire. Open ended web probes give survey methodologists the ability to collect qualitative insights across a much larger sample than is possible in traditional qualitative methods such as cognitive interviews and ethnography. In theory, this larger sample of data should provide researchers with a more nuanced understanding of the phenomenon under study given the greater geographic and demographic coverage that a larger sample provides. However, there is a risk that differential data quality across population and methodological subgroups could have negative effects on a study’s findings, and systematic analyses of the quality of open-ended probe responses by survey panelists is needed. This presentation will report on the National Center for Health Statistics’ (NCHS) current efforts to remedy this gap.
Using data from multiple rounds of NCHS’ Research and Development Survey system, which uses both statistically sampled and opt-in panels to conduct methodological research and question evaluations, this presentation will explore differences in open-ended data quality. Specifically, we explore differences based on educational attainment, race and ethnicity, age, panel type, and panel tenure through analysis of item non-response, text length, and text content. Our results suggest that there are meaningful differences in qualitative data quality across groups of respondents in survey panels. As a result, qualitative analyses based on these data should document this differential data quality and take it into account when producing findings.
Comparability of the WVS Acceptance of Homosexuality Measure: Revealing Respondents’ Associations and Reasoning
Dr Katharina Meitinger (Utrecht University) - Presenting Author
Dr Angelo Morreti (Utrecht University)
Mr Shumao Yu (Utrecht University)
One of the most frequently used measures of acceptance of lesbians and gays is the World Value Survey Acceptance of Homosexuality measure that asks respondents on a scale from 1 (“Never justifiable) to 10 (“Always”) to indicate whether they think that homosexuality can always be justified, never be justified, or something in between.”
However, several issues potentially challenge the validity and comparability of this measure: 1) The term “homosexuality” is vague and could refer to lesbians or gay men and this understanding could differ across countries. 2) The term “justified” refers to a moral logic and does not match an attitudinal question, thus, some respondents might struggle with it. 3) Respondents in different countries might assign various meanings and reasonings to each scale point.
As part of a NWO funded research project on cross-national measures of attitudes towards LGBT, we conducted a web probing study in six countries (France, Germany, Italy, the Netherlands, Poland, USA) in 2024. After the World Value Survey Acceptance of Homosexuality measure, respondents received a category selection probe asking them why they selected a specific response category.
This presentation discusses the validity and comparability of the World Value Survey Acceptance of Homosexuality measure and will present similarities and differences of associations and reasoning across the six countries. It will particularly focus on several issues that the qualitative insights from web probing revealed, such as struggle with the term “justified,” silent misinterpretations, or the too broad scope of the question.
Understanding and preventing unwanted context effects in probe response content: The challenges of pretesting related survey questions
Dr Patricia Hadler (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Ms Dörte Naber (Universidad de Granada)
Professor José Luis Padilla (Universidad de Granada)
Cognitive pretesting is used to gain insights on respondents’ thought processes when answering survey questions and to assess the validity of survey items. Many studies have examined the impact of web probing design on the quality of probes responses in terms of “hard measures”, such as the share of interpretable answers or number of themes. In this paper, we focus on systematic effects of web probing design on probe response content that point to reduced response quality. Specifically, we look at
(1) probe contamination, that is when themes from previous survey questions are erroneously included in probe response content (“spill-over effects”) and
(2) signs of perceived redundancy, that is when respondents falsely believe that different probes are asking them the same question.
Both of these issues are most likely to occur when two or more survey questions are topically related. In this talk, we present findings from a randomized 2x2 web experiment with a total of 4,000 respondents in Germany, in which pairs of related survey questions were randomized in terms of question order (A-B, B-A) and probe placement (concurrent, retrospective). In total, we used four survey question pairs referring to either general-specific or attitude-behavior question pairs.
We examine whether respondents are more likely to name topics related to the specific survey question (i.e., relationship satisfaction) in answer to a probe about the general question (i.e., life satisfaction) when the specific survey question is asked first or when the probes are asked retrospectively. We also examine whether respondents are more likely to perceive the second-shown probe as redundant in these settings.
The study concludes on how to face the challenges of pretesting related survey questions.
