ESRA 2025 Preliminary Program
All time references are in CEST
Representativeness in Social and Health Surveys |
Session Organisers |
Dr Olga Maslovskaya (University of Southampton) Professor Paul Smith (University of Southampton) Professor Peter Smith (University of Southampton)
|
Time | Tuesday 15 July, 13:45 - 15:00 |
Room |
Ruppert 040 |
Term “representativeness” is commonly used when results of social surveys are presented and discussed. Representativeness is important when the results of the analysis are generalised from the social and health surveys to the target populations. Representativeness might mean different things in different contexts, for example, cross-sectional and longitudinal contexts, it might also mean different things in different disciplines. However, it is of crucial importance to standardise and to use the same terminology within and across disciplines. Also, it is important to explore effective methods of assessing representativeness in different contexts and to identify requirements for when researchers can claim representativeness.
In this session we would like to discuss both: a) the definitions and conceptualisation of representativeness; and b)different methods of assessing representativeness in social and health surveys and their applications.
We encourage papers from researchers with a variety of backgrounds and across different sectors and disciplines, including academia, national statistics, and research agencies. We particularly welcome contributions that use experimental designs, and/or other designs that can inform future strategies for assessing representativeness in social and health surveys.
This session is proposed by the ESRC-funded project “Understanding Coverage in UK Longitudinal Population Studies”.
Keywords: representativeness, assessing representativeness, cross-sectional surveys, longitudinal surveys
Papers
A Comparative Study of Online Probability-Based and Opt-In Samples in Europe Using KnowledgePanel Europe.
Dr Joke Depraetere (Ipsos KnowledgePanel Europe) - Presenting Author
Dr Femke Dekeulenaer (Ipsos KnowledgePanel Europe)
Mrs Cristina Tudose (Ipsos KnowledgePanel Europe)
As public opinion research increasingly shifts towards online data collection, probability-based and opt-in panels have become prevalent methods for online surveys. However, their methodologies and industry practices differ significantly. This research analyzes four distinct samples across three European nations—Sweden, France, and the Netherlands—to explore the data accuracy of current online sampling approaches. For each country, data includes a probability-based sample from KnowledgePanel Europe (Ipsos’ online random probability panel, operating in the US since 1999, the UK since 2020, and expanding across Europe since 2022) and three opt-in samples.
This research assesses the accuracy of each sample against several benchmark variables derived from established random probability surveys. These benchmarks encompass diverse measures spanning health, living situations, political views, and safety & discrimination. By comparing the performance of probability-based and opt-in samples across these variables and countries, this study offers insights into the strengths and weaknesses of each approach within the European context. The findings contribute to a deeper understanding of data accuracy and potential biases associated with different online sampling methods, informing researchers and practitioners in their selection of appropriate survey methodologies.
Furthermore, the study's European focus adds to the existing literature, which predominantly examines US samples, providing a broader perspective on the generalizability of online sampling methods. The use of a comprehensive set of benchmark variables allows for a nuanced evaluation of sample accuracy across a range of socio-political indicators.
Reaching Consensus on Representativeness and Coverage in Longitudinal Population Studies: a Delphi Study
Professor Nisreen Alwan (University of Southampton) - Presenting Author
Professor Ann Berrington (University of Southampton)
Mr Aaron Vanam (University of Southampton)
Dr Olga Maslovskaya (University of Southampton)
Professor Janis Baird (University of Southampton)
Professor Nicholas Harvey (University of Southampton)
Mr Francesco Pantalone (University of Southampton)
Professor Peter Smith (University of Southampton)
Professor Paul Smith (University of Southampton)
Background: The UK possesses a large collection of longitudinal data utilised by researchers from multiple disciplines. However, there are no universal agreed definitions of what is meant by terms such as sample ‘coverage’, ‘representativeness’ and ‘external validity’ that are consistently applied across disciplines. We aimed to achieve consensus among experts across the disciplines of survey methodology, demography, statistics, epidemiology and public health on the meaning and use of such concepts.
Method: This study used the Delphi method. First, a literature review identified key issues around the subject. A series of Delphi statements were conceptualised by the research team and fielded in round one. The experts were asked to agree or disagree with proposed statements under eight sections with the opportunity to add free text. Based on the responses received, a follow-up questionnaire was crafted with 23 statements, further refining them from the first round in light of the feedback. A final third iteration of the Delphi statements was fielded and consensus achieved on the majority of items.
Results: A total of 86 experts were invited. These experts were mostly from the UK and a few were from Europe and North America. 35 experts completed the first round with agreement on 16 statements, 33 the second round with agreement on a further 6 statements and 26 the third round (response rates 41%, 85% and 93% respectively). By the end of the third round consensus (agreement or disagreement >70%) was reached on 29 out of the 34 statements.
Conclusion: Consensus was reached across a range of disciplines as to the meaning of the terms coverage and representativeness. The term ‘external validity’ appears to have different meanings across disciplines and consensus around this term was harder to reach. Our findings are likely to help study design and science communications across health and
Concepts and Measures of Coverage and Representativeness
Professor Paul Smith (Dept of Social Statistics & Demography, University of Southampton) - Presenting Author
Dr Olga Maslovskaya (Dept of Social Statistics & Demography, University of Southampton)
Dr Francesco Pantalone (Dept of Social Statistics & Demography, University of Southampton)
Professor Peter Smith (Dept of Social Statistics & Demography, University of Southampton)
Dr Mengting Zhu (Dept of Social Statistics & Demography, University of Southampton)
Professor Nisreen Alwan (School of Primary Care, Population Sciences and Medical Education, University of Southampton)
Professor Ann Berrington (Dept of Social Statistics & Demography, University of Southampton)
Professor Janis Baird (MRC Lifecourse Epidemiology Centre, University of Southampton)
Professor Nicholas Harvey (MRC Lifecourse Epidemiology Centre, University of Southampton)
We examine the use of the concepts of “coverage” and “representativeness” as presented in the literature relating to surveys and similar data sources, concentrating on examples from large-scale data assets with intended wide coverage. The motivation for this study is the investigation of the concept of coverage in UK longitudinal data sources, involving case studies with UK Biobank and the British Cohort Study (BCS70).
We reviewed the literature on coverage and representativeness in surveys, and examined both how various different definitions have been constructed and used, and how publications use these terms in a vague way. We discuss whether these are concepts which belong to the data (in general) or to analyses of the data (often specific to particular research questions). We discuss a range of possible metrics for coverage and representativeness, particularly R-indicators and comparison with benchmarks, and the practical and methodological challenges with applying them. We illustrate their application with the case studies, and draw out common themes and differences.
We present a toolkit for assessing coverage and representativeness, intended to support a range of uses, from survey commissioning to data analysis, and with particular emphasis on longitudinal data resources.