ESRA 2025 Preliminary Program
All time references are in CEST
Assessing and improving survey data quality in low- and middle-income countries (LMICs) 2 |
| Session Organisers |
Professor Timothy Johnson (University of Illinois at Chicago)
Mrs P. Linh Nguyen (University of Essex, University of Mannheim)
Dr Yfke Ongena (University of Groningen)
|
| Time | Tuesday 15 July, 15:30 - 17:00 |
| Room |
Ruppert rood - 0.51 |
Researchers working in low- and middle-income countries from diverse disciplines, such as development economics, demography, and other social sciences, are increasingly engaged in investigating different aspects of the Total Survey Error to improve data quality. They are especially concerned by how survey data quality affects substantial results, as well as poverty and demographic rates.
This session aims to mainstream research on all aspects of survey data quality stemming from LMICs where the historical development, the conditions, and the implementation of survey methodology differs from other contexts. We see this session as unique opportunity to foster the network of like-minded researchers and practitioners, as well as to promote research results focusing on LMICs in preparation for the ESRA conference in 2027 organised jointly with the World Association for Public Opinion Research (WAPOR) and the first WAPOR conference in a Sub-Saharan African country, Kenya, in 2028.
Researchers may present their work on any issue(s) encountered along the full survey lifecycle from questionnaire development and testing, including scale development; translation, adaptation, and assessment of questionnaires into local languages; sampling innovations using unconventional sample frames; survey participation, data collection challenges and solutions through innovative uses of technology; minimizing measurement error; interviewer effects; survey data quality control; respondent comprehension and burden; etc.
There is no specific regional focus and papers may cover a variety of topics. Nevertheless, the studies to be considered should rely on data coming from LMICs. Cross-national comparisons in these contexts are also welcome.
Keywords: cross-cultural survey methods
Papers
Using paradata to improve face-to-face data collection
Mr Blake Zachary (The Demographic and Health Surveys (DHS) Program) - Presenting Author
Monitoring data quality in face-to-face interviews is difficult but has been made easier through computer-assisted interviewing. In addition to answers recorded, we can now look at data about the data collection process (paradata). The USAID-funded Demographic and Health Surveys (DHS) Program supports local governments in low and middle-income countries to conduct household and facility-based surveys. We recently began routinely collecting paradata using CSPro. What can paradata tell us about the implementation of data collection, data quality, and opportunities for improvement?
Using preliminary internal data, we have calculated the average time interviewers spent on each question and compared that to a threshold of time to read the question. This comparison allows us to flag questions that take longer or shorter than expected. However, there are many reasons why the assumed timing would not equal the actual including issues of translation and respondent comprehension. These nuances need to be explored further.
The paradata also log the appearance of every error message displayed to interviewers. Tabulating these by question can give us insight into areas where either interviewers or respondents struggle to answer the question within the constraints given (for example, consistency between date of birth and current age). Reviewing the types and number of error messages can allow us to target our training approach.
Finally, new systems deployed can be quantitatively measured. In collecting dietary recall information, interviewers can ask about each food group or use a menu system to tick off multiple food items as the respondent recalls them. By tabulating the frequency in which this new menu system is used, we can modify the approach and related training in future surveys.
Using paradata we can improve monitoring of data quality during fieldwork, refine face-to-face data collection methods, and target training efforts.
Gender Role Attitudes In The Philippines And Elsewhere
Professor Harry Ganzeboom (VU University Amsterdam) - Presenting Author
The Gender Role Attitude (GRA) index is a 5-11 indicator instrument to measure differences in opinions about the proper division of household tasks. The instrument has been used in ISSP’s Gender&Family modules since 1988. Using a multiple-correspondence analysis, the ISSP1994 data on GRA were critically examined by Blasius (2006), who scorns particularly the Philippine data for being "completely inconsistent", and concludes that “it makes no sense to compare the data (…) from the Philippines with that of any of the other countries”. We take up the challenge by examining the GRA data from a broader and less technical design. First, we analyse measurement quality in all four ISSP Gender&Family modules, which brings a considerably broader database of countries and time-points. Second, we examine measurement validity not by internal associations of the instrument, but in relationship to external validation criteria, such as gender and cohort. Third, we also seek to separate validity from reliability and test whether an instrument with poor (but some) reliability can still bring out structural relationships with the validation criteria.
Our first finding is that the Philippine GRA-data indeed have low reliability but are far from randomly generated. This low reliability is consistent across waves and is replicated for a number of ISSP countries that are similar either in socio-economic development or geographical/cultural location: Brazil, Mexico, Thailand, India, Japan, China, Taiwan and Venezuela. The problem may be more substantive than technical. Second, we find that despite low reliability, the cross-national ranking of the Philippines as very gender-role conservative is consistent between ISSP waves, which confirms that data with low reliability can still contain substantive meaning. Third, we find that the usual individual determinants of differences work in the Philippines just the same as elsewhere.
Sex and the Survey: A Cross-national Comparison of Surveys Questions on Sexual and Reproductive Health and Rights
Dr Joe Strong (Queen Mary University of London) - Presenting Author
Dr Heini Väisänen (French Institute for Demographic Studies (INED))
Mrs P. Linh Nguyen (French Institute for Demographic Studies (INED), University of Essex, University of Mannheim)
Nationally representative surveys are a critical tool for producing data that can be used to measure progress across a range of sexual and reproductive health and rights (SRHR) indicators. The questions eliciting SRHR data are the result of a range of decisions, definitions, and assumptions setting the framework for survey data quality. However, how and what questions are asked varies across contexts, highlighting that SRHR data remain deeply political and the product of several assumptions.
We interrogate the similarities and differences between SRHR questions through a cross-contextual analysis of nationally representative surveys in four countries: France (Baromètre de Santé), Ghana (DHS, PMA2020), Senegal (DHS), and the United Kingdom (Natsal-3). Country selection was purposive to allow for comparison between Global North and South and anglophone and francophone surveys. Surveys were deemed relevant if they were used to measure two separate indicators relating to SRHR, per the World Development Indicators. If the survey was part of a series, all surveys within that series between the years 2010 and 2019 were included for analysis. Operationalising the Guttmacher-Lancet definition of SRHR, we iteratively developed a codebook extracting relevant questions from all surveys for thematic analysis.
Our preliminary findings highlight critical differences between surveys in the Global North and South. Surveys in the Global North incorporated concepts of pleasure, consent, and satisfaction, as well as more expansive questions relating to different types of sex. By contrast, surveys in the Global South were focused on biomedical components of SRHR and assumed sex to be penile-vaginal. This included a focus on contraceptive use and the contexts in which contraceptives are not being used. Such assumptions minimise the capacity for policies and programmes to meet the holistic SRHR needs of a population and continue to pathologise sex and reproduction in the Global South.
