ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Assessing and improving survey data quality in low- and middle-income countries (LMICs) 3
Session Organisers	Professor Timothy Johnson (University of Illinois at Chicago) Mrs P. Linh Nguyen (University of Essex, University of Mannheim) Dr Yfke Ongena (University of Groningen)
Time	Wednesday 16 July, 09:00 - 10:30
Room	Ruppert rood - 0.51

Researchers working in low- and middle-income countries from diverse disciplines, such as development economics, demography, and other social sciences, are increasingly engaged in investigating different aspects of the Total Survey Error to improve data quality. They are especially concerned by how survey data quality affects substantial results, as well as poverty and demographic rates.

This session aims to mainstream research on all aspects of survey data quality stemming from LMICs where the historical development, the conditions, and the implementation of survey methodology differs from other contexts. We see this session as unique opportunity to foster the network of like-minded researchers and practitioners, as well as to promote research results focusing on LMICs in preparation for the ESRA conference in 2027 organised jointly with the World Association for Public Opinion Research (WAPOR) and the first WAPOR conference in a Sub-Saharan African country, Kenya, in 2028.

Researchers may present their work on any issue(s) encountered along the full survey lifecycle from questionnaire development and testing, including scale development; translation, adaptation, and assessment of questionnaires into local languages; sampling innovations using unconventional sample frames; survey participation, data collection challenges and solutions through innovative uses of technology; minimizing measurement error; interviewer effects; survey data quality control; respondent comprehension and burden; etc.

There is no specific regional focus and papers may cover a variety of topics. Nevertheless, the studies to be considered should rely on data coming from LMICs. Cross-national comparisons in these contexts are also welcome.

Keywords: cross-cultural survey methods

Papers

Challenges Conducting Cognitive Interviews in Low and Middle Income Countries: A Case Study with Older Adults in Lebanon

Ms Mayssan Kabalan (American University of Beirut)
Ms Alexandra Abi Nassif (American University of Beirut)
Dr Carlos Mendes de Leon (Georgetown University)
Ms Julie de Jong (ICF International)
Dr Frederick Conrad (University of Michigan) - Presenting Author

The global increase in population ageing, especially in Low and Middle Income Countries (LMIC), has led to a growing number of new population-based surveys that document the health and social needs of older adults in these settings. These survey studies often rely on questions and instruments adapted from surveys in high-income countries, hence the importance of pretesting these surveys using methods such as cognitive interviewing to evaluate linguistic and cultural appropriateness as well as question validity. The aim of this paper is to highlight important challenges in the administration of cognitive interviews with older adults in LMIC settings. We analysed cognitive interviews with 58 older adults who participated in a pretest of a new population survey study in Lebanon. We identified two sets of challenges: population-specific challenges and ageing-specific challenges. The first set includes trouble understanding the purpose of cognitive interviewing, respondents’ suspicion of institutions, reluctance to disclose sensitive information, and problems arising from diglossia, i.e., languages whose formal, written form differs substantially from the colloquial, spoken form. The second set concerns age-related attributes such as hearing loss, fatigue, and decline in cognitive abilities. Based on our findings and experience, we offer recommendations and best practices for conducting cognitive interviews in LMIC with older adults.

Effects of interviewer experience and training on survey data quality

Dr Luke Plutowski (LAPOP Lab, Vanderbilt University) - Presenting Author

Past research and conventional wisdom suggest that more experienced interviewers produce better data than less experienced interviewers in face-to-face surveys. In this paper, we argue that prior experience is far less important than study-specific knowledge and training for ensuring data quality. We examine data from a cross-national public opinion study of 22 low- and middle-income countries, with information on more than 1000 interviewers. Results show that interviewer “skill”, as measured by performance on pre-fieldwork certification quiz on study protocol, consistently predicts higher interview quality on a number of metrics, most notably a comprehensive 0-100 quality control (QC) score. On the other hand, self-reported experience with other public opinion surveys does not necessarily lead to higher QC scores. In some cases, past experience actually negatively correlates with desired outcomes. We speculate that prior experience may lead interviewers to confuse guidelines between projects and to use “tricks” that violate the study’s protocols, leading to lower QC scores. The results underscore the need for complete and regular training for interviewers. Further, the finding that novice interviewers can produce reliable data with proper training offers positive news for researchers working in contexts without a professional survey industry or large pools of experienced survey-takers.

Adjusting for population distributions in fast changing environments: Weighing strategies using remotely sensed data in greater Nairobi and Kampala areas

Mr Pierre Schlegel (Université d'Aix Marseille / INED) - Presenting Author
Dr Valérie Golaz (Université d'Aix Marseille / INED)
Dr Basile Rousse (Université Paris Cité / INED)
Mrs Helen Habib (APHRC)
Dr Ramatou Ouedraogo (APHRC)
Dr Charles Lwanga (Makarere University)
Dr Stephen Wandera (Makarere University)
Mr Jude Otim (Makarere University)
Mr Philip Abughul (Université de Genève (UNIGE))
Professor Clémentine Rossier (Université de Genève (UNIGE))

Sampling and weighting processes are essential steps of a data collection operation that ensure the representativity of the collected data. An up-to-date description of the spatial distribution of the population as well as other basic characteristics, such as the age and sex distribution, serve as references to adjust the collected data to the population studied. When such up-to-date information is not available, these tasks are generally based on recent census data, that provide an accurate spatial distribution of the population. However, in many LMICs, this approach is hindered by three phenomena: first, national sampling frames are not updated regularly. Second, censuses are scarcer, the most recent ones sometimes dating more than 5 years. Third, population growth and rapid urbanisation strongly impact population distribution and densities. As a result, the available population distributions are outdated and do not represent current imbalances in population distributions.
However, satellite data are now freely available at a fine resolution and provide an alternative way of calculating weights. In this paper, we present innovative strategies for weighting data in the context of fast changing territories, on the basis of the data collected in July and August 2024 in FamilEA (The remaking of the Family in East Africa, a SNF funded research programme). The quantitative part of FamilEA is a survey conducted on 2000 adults aged 18 to 64 years, in each of the extended cities of Kampala and Nairobi.
This communication aims at retracing the different weightings strategies that were tried, and discuss their accuracy, to offer reflexions on weighting strategies in developing urban contexts. For this purpose, beyond the provided household counts, we used satellite data based sources such as the WorldPop 100m datasets for 2020, or the Google Open Building temporal dataset 2016-2023, adjusted with population distributions and growth.

Improving the Quality of Telephone Survey Data in LMICs: Lessons Learned from a Panel Study in Malawi and Zambia During the COVID-19 Pandemic

Dr Erica Ann Metheney (Governance and Local Development Institute, University of Gothenburg) - Presenting Author

Telephone-based surveys face challenges such as sample bias regardless of where they are implemented, but the challenges faced are exacerbated in lower-middle income countries (LMICs) by reduced levels of phone ownership/access, poor network quality, and greater difficulty paying phone fees. However, there are times when telephone surveys are the only viable option due to safety issues, as was the case during the COVID-19 pandemic. We present the challenges encountered and the lessons learned during the fielding of two telephone panel surveys in Malawi and Zambia during the COVID-19 pandemic. We provide information on phone access and usage culture in these contexts and describe how these insights were incorporated into our call-back procedures and enumerator training to help retain respondents between rounds of the survey. We also discuss the unique sampling frame utilized by the panel survey, which was derived from the 2019 Local Governance Performance Index (LGPI) Survey, a large face-to-face household survey which used a spatial probability sampling scheme that ensured the inclusion of individuals in hard-to-reach populations. We explore how this approach to sample frame creation helped mitigate some of the common issues of telephone surveys implemented in these contexts. Drawing on the 2019 LGPI and COVID-19 panel survey data, as well as interview-level enumerator feedback, this case study serves as an example of how strategic project linking and the integration of context into procedures can help improve the quality of data collected via telephone in LMICs.