All time references are in CEST
Representativeness in Social and Health Surveys |
|
| Session Organisers | Dr Olga Maslovskaya (University of Southampton) Professor Paul Smith (University of Southampton) Professor Peter Smith (University of Southampton) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Term “representativeness” is commonly used when results of social surveys are presented and discussed. Representativeness is important when the results of the analysis are generalised from the social and health surveys to the target populations. Representativeness might mean different things in different contexts, for example, cross-sectional and longitudinal contexts, it might also mean different things in different disciplines. However, it is of crucial importance to standardise and to use the same terminology within and across disciplines. Also, it is important to explore effective methods of assessing representativeness in different contexts and to identify requirements for when researchers can claim representativeness.
In this session we would like to discuss both: a) the definitions and conceptualisation of representativeness; and b)different methods of assessing representativeness in social and health surveys and their applications.
We encourage papers from researchers with a variety of backgrounds and across different sectors and disciplines, including academia, national statistics, and research agencies. We particularly welcome contributions that use experimental designs, and/or other designs that can inform future strategies for assessing representativeness in social and health surveys.
This session is proposed by the ESRC-funded project “Understanding Coverage in UK Longitudinal Population Studies”.
Keywords: representativeness, assessing representativeness, cross-sectional surveys, longitudinal surveys
Dr Joke Depraetere (Ipsos KnowledgePanel Europe) - Presenting Author
Dr Femke Dekeulenaer (Ipsos KnowledgePanel Europe)
Mrs Cristina Tudose (Ipsos KnowledgePanel Europe)
As public opinion research increasingly shifts towards online data collection, probability-based and opt-in panels have become prevalent methods for online surveys. However, their methodologies and industry practices differ significantly. This research analyzes four distinct samples across three European nations—Sweden, France, and the Netherlands—to explore the data accuracy of current online sampling approaches. For each country, data includes a probability-based sample from KnowledgePanel Europe (Ipsos’ online random probability panel, operating in the US since 1999, the UK since 2020, and expanding across Europe since 2022) and three opt-in samples.
This research assesses the accuracy of each sample against several benchmark variables derived from established random probability surveys. These benchmarks encompass diverse measures spanning health, living situations, political views, and safety & discrimination. By comparing the performance of probability-based and opt-in samples across these variables and countries, this study offers insights into the strengths and weaknesses of each approach within the European context. The findings contribute to a deeper understanding of data accuracy and potential biases associated with different online sampling methods, informing researchers and practitioners in their selection of appropriate survey methodologies.
Furthermore, the study's European focus adds to the existing literature, which predominantly examines US samples, providing a broader perspective on the generalizability of online sampling methods. The use of a comprehensive set of benchmark variables allows for a nuanced evaluation of sample accuracy across a range of socio-political indicators.
Professor Nisreen Alwan (University of Southampton) - Presenting Author
Professor Ann Berrington (University of Southampton)
Mr Aaron Vanam (University of Southampton)
Dr Olga Maslovskaya (University of Southampton)
Professor Janis Baird (University of Southampton)
Professor Nicholas Harvey (University of Southampton)
Mr Francesco Pantalone (University of Southampton)
Professor Peter Smith (University of Southampton)
Professor Paul Smith (University of Southampton)
Background: The UK possesses a large collection of longitudinal data utilised by researchers from multiple disciplines. However, there are no universal agreed definitions of what is meant by terms such as sample ‘coverage’, ‘representativeness’ and ‘external validity’ that are consistently applied across disciplines. We aimed to achieve consensus among experts across the disciplines of survey methodology, demography, statistics, epidemiology and public health on the meaning and use of such concepts.
Method: This study used the Delphi method. First, a literature review identified key issues around the subject. A series of Delphi statements were conceptualised by the research team and fielded in round one. The experts were asked to agree or disagree with proposed statements under eight sections with the opportunity to add free text. Based on the responses received, a follow-up questionnaire was crafted with 23 statements, further refining them from the first round in light of the feedback. A final third iteration of the Delphi statements was fielded and consensus achieved on the majority of items.
Results: A total of 86 experts were invited. These experts were mostly from the UK and a few were from Europe and North America. 35 experts completed the first round with agreement on 16 statements, 33 the second round with agreement on a further 6 statements and 26 the third round (response rates 41%, 85% and 93% respectively). By the end of the third round consensus (agreement or disagreement >70%) was reached on 29 out of the 34 statements.
Conclusion: Consensus was reached across a range of disciplines as to the meaning of the terms coverage and representativeness. The term ‘external validity’ appears to have different meanings across disciplines and consensus around this term was harder to reach. Our findings are likely to help study design and science communications across health and
Mrs Martha McRoy (NORC at the University of Chicago) - Presenting Author
Mr Ned English (NORC at the University of Chicago)
Mr Ben Reist (NORC at the University of Chicago)
Ms Amie Conley (NORC at the University of Chicago)
Ms Samantha Saini (Illinois Department of Public Health)
Ms Kelsey Cutler (Illinois Department of Public Health)
NORC is partnering with the Illinois Department of Public Health (IDPH) to conduct the Healthy Illinois Survey, a multi-mode address-based (ABS) study similar to the Behavioral Risk Factor Surveillance System (BRFSS). The Healthy Illinois (HIL) Survey is conducted annually and is designed to examine a broad set of health topics, including access to health services, chronic health conditions, diet, substance abuse, and exposure to violence. One notable feature of HIL is that it will generate estimates for every county, suburban Cook County municipality, Chicago Community area, and ZIP code groupings in areas supported by population density. Consequently, the study will provide higher-resolution data than have been previously available to study topics such as health equity and social determinants of health throughout Illinois.
The survey uses a sequential mixed-mode design, allowing respondents to complete the survey on the web, on the phone through inbound CATI, or via a self-administered paper questionnaire. In our paper, we examine the representativeness of participants who responded to different modes, but also explore mode preference by respondents. That is, those that prefer (and choose) a particular mode when offered multiple options. Expanding on the mode preference literature, we use multinomial models to predict mode choice using demographics and other indicators including measures related to mode access, external distractions, and cognitive abilities. Because of the size and scope of the Healthy Illinois Survey, we are also able to explore the impact of neighborhood characteristics on mode preference. Finally, as HIL is an annual survey, we compare our models created using the inaugural 2024 data to our first waves of the 2025 data collection for accuracy. Our results will be useful to health researchers as well as practitioners of multi-mode ABS designs.
Professor Paul Smith (Dept of Social Statistics & Demography, University of Southampton) - Presenting Author
Dr Olga Maslovskaya (Dept of Social Statistics & Demography, University of Southampton)
Dr Francesco Pantalone (Dept of Social Statistics & Demography, University of Southampton)
Professor Peter Smith (Dept of Social Statistics & Demography, University of Southampton)
Dr Mengting Zhu (Dept of Social Statistics & Demography, University of Southampton)
Professor Nisreen Alwan (School of Primary Care, Population Sciences and Medical Education, University of Southampton)
Professor Ann Berrington (Dept of Social Statistics & Demography, University of Southampton)
Professor Janis Baird (MRC Lifecourse Epidemiology Centre, University of Southampton)
Professor Nicholas Harvey (MRC Lifecourse Epidemiology Centre, University of Southampton)
We examine the use of the concepts of “coverage” and “representativeness” as presented in the literature relating to surveys and similar data sources, concentrating on examples from large-scale data assets with intended wide coverage. The motivation for this study is the investigation of the concept of coverage in UK longitudinal data sources, involving case studies with UK Biobank and the British Cohort Study (BCS70).
We reviewed the literature on coverage and representativeness in surveys, and examined both how various different definitions have been constructed and used, and how publications use these terms in a vague way. We discuss whether these are concepts which belong to the data (in general) or to analyses of the data (often specific to particular research questions). We discuss a range of possible metrics for coverage and representativeness, particularly R-indicators and comparison with benchmarks, and the practical and methodological challenges with applying them. We illustrate their application with the case studies, and draw out common themes and differences.
We present a toolkit for assessing coverage and representativeness, intended to support a range of uses, from survey commissioning to data analysis, and with particular emphasis on longitudinal data resources.