Quality Frameworks and Quality Assessment in Comparative Surveys |
|
| Session Organisers |
Ms Julie de Jong (University of Michigan) Dr Kristen Cibelli Hibben (University of Michigan) |
| Time | Friday 19th July, 11:00 - 12:30 |
| Room | D19 |
Multinational, multiregional, and multicultural (3MC) surveys are designed with the goal of comparability across more than one population. To achieve comparability, these surveys need to be carefully designed according to state-of-the-art principles and standards. Numerous 3MC surveys are currently conducted within the realms of official statistics, academia, and the private sectors, and have become increasingly important to global and regional policy-making. While the success of 3MC surveys hinges on the comparability or equivalence of data across many cultures and countries, the challenges of documentation, survey quality assessment procedures and criteria are far more complex than in single-country surveys.
Discussion around survey quality and the application of various quality approaches and frameworks in 3MC surveys has increased in recent years. For example, the total survey error (TSE) framework, widely accepted as the organizing framework in the design and evaluation of single-country surveys, is being increasingly applied to 3MC surveys. Others have combined TSE along with additional approaches to survey quality including fitness for intended use and survey production process quality in integrated frameworks. In the 3MC setting, these efforts have expanded to include the concept of ‘comparison error’, which is the error introduced across each stage of a 3MC survey as well as the aggregate of error across all stages (Smith, 2011). Further, work is currently underway on an AAPOR/WAPOR task force on the quality of comparative surveys. Less attention, however, has been paid to the ways in which 3MC surveys incorporate these or other perspectives in their quality assurance and quality control frameworks or overall survey quality assessment, and what standards, if any, should be put in place for 3MC surveys.
The objective of this session is to explore the ways in which overall quality—as opposed to specific QA/QC systems or approaches—is addressed in the 3MC survey setting, and the ways in which different quality frameworks can be used to assess the extent to which different surveys address both the causes and impacts of comparison error. The first paper in the session will outline the history and theoretical framework of quality in the 3MC context, and we invite subsequent papers detailing specific approaches to overall survey quality employed in current practice by 3MC surveys.
Keywords: Comparative surveys; Survey quality; Assessment
Mr William Thorn (OECD) - Presenting Author
The objective of this session is to give an overview of the approach to quality assurance and quality control adopted in the OECD’s Programme for the International Assessment of Adult Competencies (PIAAC). PIAAC is an international comparative study that combines large scale assessment with household survey methods. The administration of an assessment of cognitive skills in three domains in a household setting in over 30 countries creates a range of challenges for the management of quality. The presentation will cover the framework that guides quality assurance and control in PIAAC, the resources' developed to ensure quality (the PIAAC Technical Standards and Guidelines, interviewer training materials and monitoring tools such as dashboards), the quality assurance processes, the assessment of data quality (data adjudication) prior to the release of data and the documentation regarding quality made available to data users in the Technical Report. Particular attention will be given to a major innovation in the 2nd Cycle of PIAAC, the development of a dashboard application that is being developed to assist both national data collection organisations and the international study team to monitor data collection and data quality in close to real time.
Dr J Kappelhof (The Netherlands Insitute for Social Research/ ESS Core Scientific Team) - Presenting Author
The European Social Survey (ESS) is an academically driven cross-national survey and aims to produce comparable, high-quality data on social structure, attitudes, values and behavior patterns in Europe. This presentation focuses on how the ESS aims to ensure optimal comparability and high quality within a cross-national context by describing the ESS approach in three key areas a) input harmonisation, b) ensuring and monitoring adherence to a uniform set of principles and procedures set out in the project specifications, and c) post hoc assessment and continues improvement. An example of the ESS approach in areas a and b involves a system of consultations, discussions and sign off procedures between the national team responsible for conducting the ESS within a particular country and relevant experts in each step of the survey life cycle. This systems allows us to pursue optimal comparability and survey quality by focusing on (inter)national best practices while also taking into account national and cultural contexts, and minimising unnecessary variations across countries wherever possible. In addition, a so-called country contact is assigned from within the ESS core scientific team to every participating country to monitor their progress throughout the entire survey life cycle and function as the main contact point for the national team as well as serve as the connective tissue between the different expert areas and the national team. With respect to area c, the country contacts are also responsible for the collection and documentation of information on key performance indicators of the different steps of the survey life cycle, and note important areas for improvement in a timely manner. Other examples of the ESS approach in the different areas are the introduction of central fieldwork monitoring system to monitor adherence during fieldwork, and a work package aimed at promoting interviewer behavior that minimizes unnecessary variation between interviewers and across countries.
Ms Sophia MacGoris (Eurofound) - Presenting Author
Eurofound, a research agency of the European Union, has been carrying out comparative cross-national EU-wide surveys at regular intervals for many years:
European Working Conditions Survey (EWCS) - 6 waves since 1990, next one in 2020
European Quality of Life Survey (EQLS), 4 waves since 2003
European Company Survey – 3 waves since 2009, next one in 2019
The surveys reflect Eurofound's commitment to answer to the needs of its tripartite stakeholders with a remit to conduct European policy-oriented research in the fields of living and working conditions. Eurofound has a strong commitment to quality standards and improvements and, considering Eurofound surveys' impact at EU, international and national levels, it is important that data collected are sound, robust and of the highest quality and that information on data quality is made available to stakeholders and the research community.
Following on from each survey, Eurofound strives to learn new lessons for further improvement with an aim to combine measuring quality with systematic activities for amelioration. For its surveys, Eurofound applies a quality assurance framework which it has developed based on the quality concept of the European Statistical System as created by Eurostat, as well as other quality frameworks such as the Cross-Cultural Survey Guidelines and the Total Survey Error Approach. During survey preparation and implementation, information is gathered to assess it against a set of quality indicators. As part of quality assurance and control, Eurofound also commissions an external data quality assessment for each survey at the end of the process. Comprehensive information on quality and methodology for all of Eurofound’s surveys are made available to the public after each one has been completed.
Ms Magali Rheault (Gallup) - Presenting Author
Ms Neli Esipova (Gallup)
Dr Anita Pugliese (Gallup)
The World Poll is a textbook case of 3MC surveys. Launched by Gallup in 2005, the World Poll conducts annual surveys in more than 140 countries of which about 100 are face-to-face surveys spanning all regions of the globe. As such, standardized processes become paramount to maximize comparability across languages and cultures. After interviewing almost 2 million world residents and more than a decade of experience in the 3MC context, the Gallup research team has developed an integrative framework to execute the World Poll for maximum comparability.
In this paper, we discuss the challenges faced to execute such a large scale research project along with lessons learned from various world regions to optimize comparability of data across time and countries. Keys to the successful implementation of the World Poll have been the 1) standardization of methods across regions, 2) centralization of processes, 3) development of a global network of research agencies 4) detailed documentation and 5) use of new technologies. While there are ongoing challenges, the Gallup research team has built a robust foundation for the consistent implementation of 3MC surveys, which it continues to improve.
Dr Kristen Cibelli Hibben (University of Michigan) - Presenting Author
Ms Julie de Jong (University of Michigan)
Multinational, multiregional, and multicultural (3MC) surveys are becoming increasingly important to global and regional decision-making and theory-building. With this has come renewed awareness of the importance of survey data quality and comparability and the challenges associated with achieving these. Various established 3MC surveys have drawn on different frameworks to guide both quality assurance and control processes as well as assessment of the quality of survey processes and outcomes. However, the approaches taken are not systematic and have often not been comprehensive. Having reviewed the status quo and associated limitations, we suggest in this paper a holistic approach to quality and quality assessment, which incorporates the most commonly used quality frameworks, including survey process quality, fitness for intended use, and Total Survey Error (TSE). The first step in this multipronged approach is the development of a checklist, used first in operations and then in subsequent assessment, to measure survey process quality. Such a checklist would include specific quality indicators linked to the widely-used quality dimensions (relevance and timeliness, accuracy, punctuality, accessibility, and coherence and comparability). The second step, guided by the concept of fitness for intended use, is the documentation and subsequent analyses of key outputs from the survey, including the final source questionnaire and the final dataset. In the third step, we turn away from the established quality frameworks and define a set of 3MC survey best practice guidelines based on both current 3MC survey practices and recent contributions to the methodological literature, and suggest that these guidelines be used to assess the a survey’s processes and outputs. The final step of our proposed quality framework draws on the TSE framework to develop a series of recommendations organized and prioritized considering both cost and relative impact or trade-offs in addressing key sources of survey error.