‘Messiness’ in Extant Cross-National Survey Data: New Approaches to Old News |
|
| Session Organisers |
Dr Irina Tomescu-Dubrow (IFiS PAN and CONSIRT) Dr Ilona Wysmulek (IFiS PAN and CONSIRT) Professor Kazimierz M. Slomczynski (IFiS PAN and CONSIRT) |
| Time | Friday 19th July, 13:30 - 14:30 |
| Room | D30 |
Cross-national survey projects exhibit wide variation in data quality, both within and across projects. Some departures from quality standards that the specialized literature has established for data collection, cleaning, and documentation, such as the presence of non-unique records (or duplicates), are unequivocal instances of ‘bad data,’ while others, such as certain types of processing errors are more ambiguous. Between the clearly bad and clearly good survey data there may be a range of ‘decent’ quality surveys, with potentially interesting and important information collected form under-surveyed countries and less well covered time periods. However, to date there is little research that systematically assesses the quality of extant international survey data, or that looks at whether and how the ‘messiness’ in existing surveys can be minimized ex-post, and with what consequences for empirical analyses.
For this session we invite theoretical and empirical papers on evaluating the quality of extant surveys, after the stages of data gathering and documentation are completed. This could include creating new metadata for survey quality. Contributions could focus on different dimensions of survey quality, including (a) the quality of surveys as reflected in the general survey documentation, (b) the degree of consistency between the description of the data with the data records in the computer files, and (c) the quality of the data records in the datasets. We aim to engage with both survey methodologists and with researchers who use survey data primarily for substantive analyses, to discuss data quality and ways of accounting for its variation in empirical analysis within and beyond the Total Survey Error, Total Survey Quality and Total Quality Management frameworks.
Keywords: Dimensions of survey quality, data quality variation, post-survey quality assessment, quality controls in substantive analyses
Dr Piotr Jabkowski (University of Poznan, Poland) - Presenting Author
This paper presents differences in sampling procedures and fieldwork execution of national surveys within five major comparative cross-country projects, i.e., Eurobarometer / Candidate Countries Eurobarometer, European Quality of Life Survey, European Social Survey, European Values Study and International Social Survey Programme. The analysis is based on a dataset describing the methodologies of 1537 surveys that have been carried out in the period from 1981 to 2017 on representative samples of the populations of European countries. The author starts with a short overview of a cross-country projects and then presents the process of methodological archivisation of sampling and fieldwork procedures. Cross-projects and long-term differences in the quality of survey documentation are also described. The main part of the presentation presents s the cross- and within-project differences in survey procedures with respect to: 1) target populations, 2) sampling frames, 3) types of survey samples and sampling designs, 4) within-household selection of target persons in address-based samples, 5) fieldwork execution (e.g., length of fieldwork, modes of data collection, presence or absence of substitutions, use of advance letters and respondents' incentives, presence of refusal conversions or back-checking procedures), and 6) fieldwork outcomes rates.
This work was supported by the grant “Comparative analysis of the quality of survey samples in the cross-national studies on the basis of external and internal criteria of representativeness: survey archivisation and meta-bases of results” awarded by the Polish National Science Centre (No. 2017/01/X/HS6/01304).
Mr Zewei Zong (SurveyMonkey) - Presenting Author
Mr Jack Chen (SurveyMonkey)
For more than a decade, millions of survey creators have used SurveyMonkey to send web surveys to their customers, employees, target markets, students etc. With access to an incredible amount of survey paradata, SurveyMonkey is in an unique position to provide a “State of Surveys” report, which gives a snapshot of surveys conducted online. In particular, the large international user base gives us a great opportunity to draw a portrait for both web survey creators and respondents in various parts of the world. What is the response rate in Philippines relative to that in the US? What time of day do respondents in India take surveys on the weekend compared to the weekdays? Do survey creators tend to use a neutral point in the answer scale in Australia? Are students in the US more sensitive to the length of surveys than those in the UK? We will answer important questions like these for the benefit of researchers conducting multinational web surveys.