All time references are in CEST
Assessing and improving survey data quality in low- and middle-income countries (LMICs) |
|
| Session Organisers | Professor Timothy Johnson (University of Illinois at Chicago) Mrs P. Linh Nguyen (University of Essex, University of Mannheim) Dr Yfke Ongena (University of Groningen) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Researchers working in low- and middle-income countries from diverse disciplines, such as development economics, demography, and other social sciences, are increasingly engaged in investigating different aspects of the Total Survey Error to improve data quality. They are especially concerned by how survey data quality affects substantial results, as well as poverty and demographic rates.
This session aims to mainstream research on all aspects of survey data quality stemming from LMICs where the historical development, the conditions, and the implementation of survey methodology differs from other contexts. We see this session as unique opportunity to foster the network of like-minded researchers and practitioners, as well as to promote research results focusing on LMICs in preparation for the ESRA conference in 2027 organised jointly with the World Association for Public Opinion Research (WAPOR) and the first WAPOR conference in a Sub-Saharan African country, Kenya, in 2028.
Researchers may present their work on any issue(s) encountered along the full survey lifecycle from questionnaire development and testing, including scale development; translation, adaptation, and assessment of questionnaires into local languages; sampling innovations using unconventional sample frames; survey participation, data collection challenges and solutions through innovative uses of technology; minimizing measurement error; interviewer effects; survey data quality control; respondent comprehension and burden; etc.
There is no specific regional focus and papers may cover a variety of topics. Nevertheless, the studies to be considered should rely on data coming from LMICs. Cross-national comparisons in these contexts are also welcome.
Keywords: cross-cultural survey methods
Mrs P. Linh Nguyen (French Institute for Demographic Studies (INED), University of Essex, University of Mannheim) - Presenting Author
Most low- and middle-income countries are multilingual leading to the situation that both interviewer and respondent are speaking multiple local languages to a varying degree. As multilingual respondents, especially those with lower education, differ in their proficiency in the survey language, some will exhibit more cognitive processing problems evidenced by audible manifestations of problematic interactional behaviours during the interview (i.e., through seeking for clarification or repetition of the question).
Using the interactional analysis of the recordings of ten selected questions in a survey on financial behaviour and attitudes in Zambia on a sample of more than 800 interviews in two local languages (Bemba and Chewa), we analyze the relationship between six indicators of problematic interactional behaviours and interviewer effects (1. language switches by either interviewers or respondents; 2. exact reading of the question; 3.any pre-emptive and follow-up behaviours by interviewers to obtain a codable answer (such as providing explanations without being asked or feedback or probing); 4. seeking clarification; 5. indicators of uncertainty (including providing pauses, fillers, and repairs, as well as verbal expressions of uncertainty).
This study’s objective is to document this process of switching language and its implication to survey data quality. The Zambian survey we rely on estimates that about 2 to 7 percent of interviews exhibit some form of language switching based on the analysis of 10 recorded and analysed questions for more than 1,000 respondents from a probabilistic household survey in Zambia. The results show for both provinces that language switching does not occur as a single phenomenon but always in co-occurrence with other problematic interactional behaviour. Thus, we can categorise language switching as another problematic behaviour indicating the breakdown of the cognitive answer process and a disruption to the ideal question-answer sequence.
Ms Mayssan Kabalan (American University of Beirut)
Ms Alexandra Abi Nassif (American University of Beirut)
Dr Carlos Mendes de Leon (Georgetown University)
Ms Julie de Jong (ICF International)
Dr Frederick Conrad (University of Michigan) - Presenting Author
The global increase in population ageing, especially in Low and Middle Income Countries (LMIC), has led to a growing number of new population-based surveys that document the health and social needs of older adults in these settings. These survey studies often rely on questions and instruments adapted from surveys in high-income countries, hence the importance of pretesting these surveys using methods such as cognitive interviewing to evaluate linguistic and cultural appropriateness as well as question validity. The aim of this paper is to highlight important challenges in the administration of cognitive interviews with older adults in LMIC settings. We analysed cognitive interviews with 58 older adults who participated in a pretest of a new population survey study in Lebanon. We identified two sets of challenges: population-specific challenges and ageing-specific challenges. The first set includes trouble understanding the purpose of cognitive interviewing, respondents’ suspicion of institutions, reluctance to disclose sensitive information, and problems arising from diglossia, i.e., languages whose formal, written form differs substantially from the colloquial, spoken form. The second set concerns age-related attributes such as hearing loss, fatigue, and decline in cognitive abilities. Based on our findings and experience, we offer recommendations and best practices for conducting cognitive interviews in LMIC with older adults.
Dr Luke Plutowski (LAPOP Lab, Vanderbilt University) - Presenting Author
Past research and conventional wisdom suggest that more experienced interviewers produce better data than less experienced interviewers in face-to-face surveys. In this paper, we argue that prior experience is far less important than study-specific knowledge and training for ensuring data quality. We examine data from a cross-national public opinion study of 22 low- and middle-income countries, with information on more than 1000 interviewers. Results show that interviewer “skill”, as measured by performance on pre-fieldwork certification quiz on study protocol, consistently predicts higher interview quality on a number of metrics, most notably a comprehensive 0-100 quality control (QC) score. On the other hand, self-reported experience with other public opinion surveys does not necessarily lead to higher QC scores. In some cases, past experience actually negatively correlates with desired outcomes. We speculate that prior experience may lead interviewers to confuse guidelines between projects and to use “tricks” that violate the study’s protocols, leading to lower QC scores. The results underscore the need for complete and regular training for interviewers. Further, the finding that novice interviewers can produce reliable data with proper training offers positive news for researchers working in contexts without a professional survey industry or large pools of experienced survey-takers.
Mr Blake Zachary (The Demographic and Health Surveys (DHS) Program) - Presenting Author
Monitoring data quality in face-to-face interviews is difficult but has been made easier through computer-assisted interviewing. In addition to answers recorded, we can now look at data about the data collection process (paradata). The USAID-funded Demographic and Health Surveys (DHS) Program supports local governments in low and middle-income countries to conduct household and facility-based surveys. We recently began routinely collecting paradata using CSPro. What can paradata tell us about the implementation of data collection, data quality, and opportunities for improvement?
Using preliminary internal data, we have calculated the average time interviewers spent on each question and compared that to a threshold of time to read the question. This comparison allows us to flag questions that take longer or shorter than expected. However, there are many reasons why the assumed timing would not equal the actual including issues of translation and respondent comprehension. These nuances need to be explored further.
The paradata also log the appearance of every error message displayed to interviewers. Tabulating these by question can give us insight into areas where either interviewers or respondents struggle to answer the question within the constraints given (for example, consistency between date of birth and current age). Reviewing the types and number of error messages can allow us to target our training approach.
Finally, new systems deployed can be quantitatively measured. In collecting dietary recall information, interviewers can ask about each food group or use a menu system to tick off multiple food items as the respondent recalls them. By tabulating the frequency in which this new menu system is used, we can modify the approach and related training in future surveys.
Using paradata we can improve monitoring of data quality during fieldwork, refine face-to-face data collection methods, and target training efforts.
Mr Pierre Schlegel (Université d'Aix Marseille / INED) - Presenting Author
Dr Valérie Golaz (Université d'Aix Marseille / INED)
Dr Basile Rousse (Université Paris Cité / INED)
Mrs Helen Habib (APHRC)
Dr Ramatou Ouedraogo (APHRC)
Dr Charles Lwanga (Makarere University)
Dr Stephen Wandera (Makarere University)
Mr Jude Otim (Makarere University)
Mr Philip Abughul (Université de Genève (UNIGE))
Professor Clémentine Rossier (Université de Genève (UNIGE))
Sampling and weighting processes are essential steps of a data collection operation that ensure the representativity of the collected data. An up-to-date description of the spatial distribution of the population as well as other basic characteristics, such as the age and sex distribution, serve as references to adjust the collected data to the population studied. When such up-to-date information is not available, these tasks are generally based on recent census data, that provide an accurate spatial distribution of the population. However, in many LMICs, this approach is hindered by three phenomena: first, national sampling frames are not updated regularly. Second, censuses are scarcer, the most recent ones sometimes dating more than 5 years. Third, population growth and rapid urbanisation strongly impact population distribution and densities. As a result, the available population distributions are outdated and do not represent current imbalances in population distributions.
However, satellite data are now freely available at a fine resolution and provide an alternative way of calculating weights. In this paper, we present innovative strategies for weighting data in the context of fast changing territories, on the basis of the data collected in July and August 2024 in FamilEA (The remaking of the Family in East Africa, a SNF funded research programme). The quantitative part of FamilEA is a survey conducted on 2000 adults aged 18 to 64 years, in each of the extended cities of Kampala and Nairobi.
This communication aims at retracing the different weightings strategies that were tried, and discuss their accuracy, to offer reflexions on weighting strategies in developing urban contexts. For this purpose, beyond the provided household counts, we used satellite data based sources such as the WorldPop 100m datasets for 2020, or the Google Open Building temporal dataset 2016-2023, adjusted with population distributions and growth.
Professor Harry Ganzeboom (VU University Amsterdam) - Presenting Author
The Gender Role Attitude (GRA) index is a 5-11 indicator instrument to measure differences in opinions about the proper division of household tasks. The instrument has been used in ISSP’s Gender&Family modules since 1988. Using a multiple-correspondence analysis, the ISSP1994 data on GRA were critically examined by Blasius (2006), who scorns particularly the Philippine data for being "completely inconsistent", and concludes that “it makes no sense to compare the data (…) from the Philippines with that of any of the other countries”. We take up the challenge by examining the GRA data from a broader and less technical design. First, we analyse measurement quality in all four ISSP Gender&Family modules, which brings a considerably broader database of countries and time-points. Second, we examine measurement validity not by internal associations of the instrument, but in relationship to external validation criteria, such as gender and cohort. Third, we also seek to separate validity from reliability and test whether an instrument with poor (but some) reliability can still bring out structural relationships with the validation criteria.
Our first finding is that the Philippine GRA-data indeed have low reliability but are far from randomly generated. This low reliability is consistent across waves and is replicated for a number of ISSP countries that are similar either in socio-economic development or geographical/cultural location: Brazil, Mexico, Thailand, India, Japan, China, Taiwan and Venezuela. The problem may be more substantive than technical. Second, we find that despite low reliability, the cross-national ranking of the Philippines as very gender-role conservative is consistent between ISSP waves, which confirms that data with low reliability can still contain substantive meaning. Third, we find that the usual individual determinants of differences work in the Philippines just the same as elsewhere.
Dr Erica Ann Metheney (Governance and Local Development Institute, University of Gothenburg) - Presenting Author
Telephone-based surveys face challenges such as sample bias regardless of where they are implemented, but the challenges faced are exacerbated in lower-middle income countries (LMICs) by reduced levels of phone ownership/access, poor network quality, and greater difficulty paying phone fees. However, there are times when telephone surveys are the only viable option due to safety issues, as was the case during the COVID-19 pandemic. We present the challenges encountered and the lessons learned during the fielding of two telephone panel surveys in Malawi and Zambia during the COVID-19 pandemic. We provide information on phone access and usage culture in these contexts and describe how these insights were incorporated into our call-back procedures and enumerator training to help retain respondents between rounds of the survey. We also discuss the unique sampling frame utilized by the panel survey, which was derived from the 2019 Local Governance Performance Index (LGPI) Survey, a large face-to-face household survey which used a spatial probability sampling scheme that ensured the inclusion of individuals in hard-to-reach populations. We explore how this approach to sample frame creation helped mitigate some of the common issues of telephone surveys implemented in these contexts. Drawing on the 2019 LGPI and COVID-19 panel survey data, as well as interview-level enumerator feedback, this case study serves as an example of how strategic project linking and the integration of context into procedures can help improve the quality of data collected via telephone in LMICs.
Dr Joe Strong (Queen Mary University of London) - Presenting Author
Dr Heini Väisänen (French Institute for Demographic Studies (INED))
Mrs P. Linh Nguyen (French Institute for Demographic Studies (INED), University of Essex, University of Mannheim)
Nationally representative surveys are a critical tool for producing data that can be used to measure progress across a range of sexual and reproductive health and rights (SRHR) indicators. The questions eliciting SRHR data are the result of a range of decisions, definitions, and assumptions setting the framework for survey data quality. However, how and what questions are asked varies across contexts, highlighting that SRHR data remain deeply political and the product of several assumptions.
We interrogate the similarities and differences between SRHR questions through a cross-contextual analysis of nationally representative surveys in four countries: France (Baromètre de Santé), Ghana (DHS, PMA2020), Senegal (DHS), and the United Kingdom (Natsal-3). Country selection was purposive to allow for comparison between Global North and South and anglophone and francophone surveys. Surveys were deemed relevant if they were used to measure two separate indicators relating to SRHR, per the World Development Indicators. If the survey was part of a series, all surveys within that series between the years 2010 and 2019 were included for analysis. Operationalising the Guttmacher-Lancet definition of SRHR, we iteratively developed a codebook extracting relevant questions from all surveys for thematic analysis.
Our preliminary findings highlight critical differences between surveys in the Global North and South. Surveys in the Global North incorporated concepts of pleasure, consent, and satisfaction, as well as more expansive questions relating to different types of sex. By contrast, surveys in the Global South were focused on biomedical components of SRHR and assumed sex to be penile-vaginal. This included a focus on contraceptive use and the contexts in which contraceptives are not being used. Such assumptions minimise the capacity for policies and programmes to meet the holistic SRHR needs of a population and continue to pathologise sex and reproduction in the Global South.
Dr Hilal Arslan (Hacettepe University Institute of Population Studies) - Presenting Author
Mrs Aslıhan Kabadayı (Hacettepe University Institute of Population Studies)
Like many countries, the COVID-19 pandemic has forced Turkish Statistical Institute (TURKSTAT) to make changes in its data collection and fieldwork strategy due to limited opportunities to conduct face-to-face interviews. Therefore, for majority of the surveys face-to-face CAPI sampled are substituted with CATI data collection mode suddenly. Based on Total Survey Error approach main types of measurement errors stemmed from data collection mode are due to nonresponse, social desirability bias, satisficing behavior, differences in handling of “don’t know” or refusal response options, contextual information, using visual or auditory cues, difficulty in attention, presence of others during interview, questionnaire design and interview length that may influence the respondent’s answers. Against this background, our study aimed to investigate the differences in the survey estimates of target variables i.e. life satisfaction, happiness and satisfaction with subdomains of life by interviewing by telephone instead of face-to-face on important target variables. Up to our knowledge this is the first study to check data quality for CAPI-CATI administrative mode transition for TURKSTAT surveys and we were particularly interested in whether these differences are due to mode-specific measurement errors including nonresponse rather than coverage errors by comparing subjective well-being indicators derived from data collected in person (CAPI) and those collected over the phone (CATI) by checking descriptive statistics and applying multivariate statistical models. For data analysis, in order to explore the potential impact of the mode difference on our survey estimates, we analyzed survey data for the years 2003-2023 by taking the complex sample design (stratification, clustering, and weighting) into account. Preliminary findings of the study show that there are statistically significant changes in the distribution of the response categories for the selected attitudinal questions and subjective measures.
Mr Glen Heller (ICF)
Mr Alexander Izmukhambetov (Data Experts Consulting International) - Presenting Author
Ms Lindsey Anna (United States Agency for International Development)
Ms Monica Kothari (ICF)
In the mid-2000s, with the decreasing cost and increasing power and availability of mobile devices, field surveys transitioned from paper-based data collection to direct capture on digital devices. This transition triggered a paradigm shift in the development of software systems to accommodate this new approach. Unlike the linear, centralized, and pipeline-like nature of paper data entry, CAPI systems need to distribute, decentralize, and scale functionality across multiple endpoints and locations. With evolving digital platforms and increasingly accessible mobile internet, it became feasible to envision, develop, and mass-adopt a robust CAPI methodology, ensuring adequate functionality, efficiency, quality, and safety.
The Surveys for Monitoring in Resilience and Food Security (SMRFS) has chosen a data management methodology that revolves around task-driven data structures. Tasks are simple data records containing information about specific activities required from the user. Each task has a well-defined lifecycle: creation, transmission, transformation, and termination. The SMRFS CAPI data management system encodes a template for generating, moving, and tracking tasks over the underlying device-based and cloud-based infrastructure. This centralizes the enforcement of workflow rules and requirements while distributing and decentralizing task execution and progression. The methodology works alongside a role-based user identity system, with each role having a well-defined set of tasks within the workflow and specified interactions with other users.
In practical terms, this methodology streamlines complex data collection operations, enforcing protocol rules and boundaries, as well as automating data management. Users on all levels are only required to review and complete tasks as they are encountered, with the system handling the rest. This reduces the time and effort needed for the technical part of interviewer training, eliminates manual workflow management, reduces human error in the field, and provides comprehensive data collection quality feedback to the survey headquarters.
Mr M Moinuddin Haider (International Centre for Diarrhoeal Disease Research, Bangladesh (icddr,b)) - Presenting Author
Mr Md Mahabubur Rahman (icddr,b)
Mr Md Tazvir Amin (icddr,b)
Dr Nurul Alam (icddrb.org)
Background: Household surveys often collect data on emotionally and socio-culturally normative and sensitive topics. Conducting interviews in the absence of third persons is recommended but ensuring privacy often becomes challenging, which may influence the responses, resulting in biased estimates. This study aims to estimate the presence of others during the interview (POODI) on survey estimates and correction factors to adjust the bias.
Method: We used the data from the Sexual and Reproductive Health Survey 2024 conducted among Forcibly Displaced Myanmar National (Rohingya Refugees) in Bangladesh. In this survey, additional data on the POODI was collected by the enumerators after each section of the questionnaire. The analytical sample includes 3205 married women aged 15-49. We examined the POODI for current contraceptive use, unintended pregnancy and birth, child death, and physical violence. We estimated the prevalence of POODI and identified its correlates using logistic regression. The correction factor was estimated using augmented inverse probability weighting that has the double-robust property. Age, education, and household size specify the logistic treatment model.
Results: The prevalence of POODI was 70%. Interestingly, the prevalence of POODI was 1.3 times higher among women who ever attended school than women without schooling. The survey underestimated contraceptive use by 4.8% and unintended pregnancy by 20% in the POODI compared to no POODI group. Reporting of unwanted birth, child death, and physical violence remained indifferent in the POODI. Post-stratification correction factors to minimize the indicator-specific underestimation will be elaborated in the main paper.
Conclusion: The high prevalence of POODI can underestimate some sensitive indicators in densely populated or refugee communities, which is also likely to happen in other conservative populations. Adding question(s) about the POODI and incorporating the correction factor may improve the estimates of socially sensitive indicators.
Dr Md Juel Rana (G. B. Pant Social Science Institute) - Presenting Author
The misreporting of date of first marriage and year of birth can influence demographic indicators such as age specific fertility rates and mortality rates. The aim of this study is to to assess the extent of quality of reporting of years of age at first marriage, and first birth using NFHS-4 data. The Whipple index of the years of first marriage and duration since first marriage have been estimated across the states of India to show the reporting of age at first marriage. Similar to the age at marriage, the completeness of the information in terms of year and month of first birth have estimated. The result shows that more than 90 percent of ever-married women reported year and month of their age at first marriage in the majority of states and union territories of India. Overall, the heaping index on years of first marriage and year of first birth ending in zero and five was 1.11 and 1.04 respectively in India. About two-third of states and UTs show some evidence of heaping on calendar years of first marriage ending in zero or five. More than 95 percent of respondents provided complete date of their first births in majority of the states and UTs except Karnataka, Telangana, Arunachal Pradesh, Andhra Pradesh, and Gujarat. A large proportion of negative first birth intervals may imply that first marriages to first birth duration has omitted or displaced. Thus, careful training of interviewers will be necessary and some inconsistencies will undoubtedly remain in the questionnaires. Field editors and supervisors should be responsible for resolving these inconsistencies. The magnitude of the data deficiencies highlighted in this paper vary between states and union territories in India.