All time references are in CEST
Falsification detection in times of crisis: Challenges, opportunities, and new directions 2 |
|
| Session Organisers |
Mr Markus Bönisch (Statistics Austria) Dr Eduard Stöger (Statistics Austria) |
| Time | Wednesday 19 July, 16:00 - 17:30 |
| Room | U6-23 |
With response rates for sample surveys continuing to decline for face-to-face surveys, incentives for interviewers may increase to help improve production, and temptations may rise for interviewers to falsify data. Methods exist to identify interviewers for which further investigation about their cases is needed. The approaches include validation of finalized cases through calling the respondents and asking them to verify the contact by the interviewer, and there are other attempts to assess the data, such as interview data, paradata, and GPS data. This session is designed to raise aspects of retaining data quality in the presence of falling response rates, in particular to detect falsification suspicions through a data driven process, to defend suspicions, and to investigate. Presentations will provide emerging approaches to detect potential falsification. The session will include a variety of surveys (e.g., Programme for the International Assessment of Adult Competencies, PIAAC), from different countries, and sectors (government, private sector, university).
Keywords: falsification, data, quality
Mr Tom Krenzke (Westat) - Presenting Author
Ms Laura Gamble (Westat)
Mrs Wendy Hicks (Westat)
The pandemic has made in-person data collection very challenging. Indications are that refusals and avoidances of interviewers are on the rise, which puts pressure on meeting sample size goals by survey organizations and interviewers. These circumstances threaten data quality and cause concerns about different components of total survey error. For example, interviewers may feel pressure to meet goals or achieve incentives or bonuses for productive work. They may be tempted to emulate the sampled household or persons’ survey responses without contact attempts. In the Programme for the International Assessment of Adult Competencies (PIAAC) in the U.S., which is sponsored by the National Center for Education Statistics, a plan has been developed and operationalised to identify interviewers who show indications of low data quality, especially where falsification is suspected. The project team has enacted three main components: validation of finalised cases through calling the respondents and asking them to verify the contact by the interviewer; use of GPS data; and use of paradata and survey data. The focus of the presentation is the implementation of the statistical methodology that uses paradata and survey data summarised at the interviewer level. We will describe the mechanics that include the real-time data flow, standardisation and distributions of key performance indicators, and the operational use of the statistical approach toward confirming falsification.
Ms Natascha Massing (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Ms Anouk Zabal (GESIS - Leibniz Institute for the Social Sciences)
Dr Britta Gauly (GESIS - Leibniz Institute for the Social Sciences)
Dr Sanja Kapidzic (GESIS - Leibniz Institute for the Social Sciences)
Ms Silke Martin (GESIS - Leibniz Institute for the Social Sciences)
Dr Désirée Niessen (GESIS - Leibniz Institute for the Social Sciences)
PIAAC, the Programme for the International Assessment of Adult Competencies, measures key skills of adults (16-65 years) in 31 countries. The PIAAC interview is administered face-to-face and consists of a comprehensive background questionnaire and a cognitive assessment. As a comparative survey that informs policy makers, the PIAAC data collection adheres to very comprehensive international standards and practices to achieve the highest possible data quality. Quality control of interviewers’ work is a crucial element and includes not only detecting potentially fabricated interviews, but also falsifications in the sense of shortcutting or other intentional departures from the PIAAC protocols. Because circumstances and biographies are so varied, there may be a fine line between unexpected but real (i.e., unfalsified) instances, and impermissible deviations or fabrications.
This presentation will describe the general validation strategy and different methods used to control interviewers’ work in Germany in the 2022/2023 PIAAC data collection. One recurring topic over all quality control methods is to determine what is plausible and what is not. This requires pooling information from all sources (e.g., register data, phone and mail validation of interviews, audio-recordings of interviews, or plausibility checks of background questionnaire responses for short biographies), checking for patterns at the interviewer level, and coming to an overall evaluation. This contribution will conclude by reflecting on the lessons learnt.
Mr Felix Deichmann (Statistics Austria) - Presenting Author
Dr Eduard Stöger (Statistics Austria)
Mr Markus Bönisch (Federal Ministry of Education, Science and Research)
In face-to-face surveys interviewers may be incentivized to falsify data, particularly when they are paid on a per piece basis. To ensure data quality it is often required to validate randomly selected interviews by contacting the respondent. In addition, it is advised to implement different methods to detect possible falsifications and have those cases validated in particular.
PIAAC (Programme for the International Assessment of Adult Competencies) consists of two parts, a background questionnaire conducted by the interviewer and a self-administered skills-assessment by the respondent. In Austria, we closely monitor finalized interviews using a composition of data driven indicators to flag potential falsifications and conspicuous interviewers. Flagged cases are then validated via email, telephone or by mail. Moreover, interviewers are confronted with suspicious cases directly after completing them. Flag indicators include the duration of administration of different sections of the survey, variation of response patterns of item batteries, skip-rate of assessment items and proportion of different reasons for early termination.
The paper will describe the falsification detection methods used for the PIAAC main survey in Austria and will show empirical evidence about what has worked well to identify falsifying interviewer.