All time references are in CEST
Making surveys more inclusive and equitable: strategies, actions and challenges |
|
| Session Organisers | Dr Véronique SIEGLER (Office for National Statistics (ONS)) Line Knudsen (National Centre for Social Research (Natcen)) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
In today's diverse society, ensuring inclusivity in social surveys is essential for generating accurate, representative, and meaningful insights that can shape policies. Without a comprehensive approach to inclusivity, there is a risk of omitting important voices and producing skewed findings. Additionally, inclusivity carries an ethical responsibility to ensure that all groups, particularly marginalised ones, are heard. This session will explore how researchers can adopt a holistic approach to inclusivity across all stages of the survey process, from respondent engagement , sampling and questionnaire design to data analysis requirements. Key to this discussion is the concept of intersectionality—people’s experiences are shaped by multiple, interconnected factors, and ignoring this complexity can lead to oversimplified approaches to inclusivity.
Barriers to inclusivity in social surveys are wide-ranging, including accessibility issues, cultural insensitivity, technological challenges, and lack of trust amongst certain communities or disenfranchised individuals. The aim of this session is to reflect on how to overcome these barriers, to share successes, challenges and lessons learned. The session will provide a platform for researchers to present innovative strategies and approaches that ensure inclusivity, contributing to a more equitable future for social surveys.
We invite papers that address a variety of topics, including but not limited to:
• Inclusivity for individuals experiencing physical or mental health issues, or neurodiversity.
• Overcoming demographic or identity-based barriers.
• Engaging hard-to-reach groups.
• Addressing the complexities of intersectionality in social surveys.
• Building trust in social surveys.
• Leveraging alternative data to address inclusivity challenges.
• Inclusivity in the context of non-traditional data sources.
We particularly welcome submissions from researchers across different sectors—academia, national statistics, and private research organisations. This session aims to inform future strategies and best practices for enhancing inclusivity in social surveys.
Keywords: inclusivity, equity, accessibility, intersectionality, hard-to-reach, trust, barriers, alternative data
Miss Molly Dawson (Ipsos) - Presenting Author
The GP Patient Survey (GPPS) is a crucial source of information for understanding patient experiences within the National Health Service (NHS) in England. This national survey is sent to around 2.5 million patients aged 16 or over and plays a vital role in informing policy and improving primary healthcare services. However, to ensure its findings reflect the population, it is essential that the survey is as accessible as possible. This abstract explores the accessible design of the GPPS, highlighting how these contribute to inclusivity and representation within social surveys.
Given the survey's scale and push-to-web methodology, ensuring inclusivity for all invitees presents challenges. A proactive approach is taken by embedding accessibility within the survey's design. This includes:
• Using clear and simple language, validated by cognitive testing with patients, consultation with lived experience partners and the Plain English Campaign's Crystal Mark standard (https://www.plainenglish.co.uk/services/crystal-mark/7-the-crystal-mark-standard.html).
• Offering diverse formats: 14 additional languages, Braille, large print, British Sign Language videos, and an accessible online platform compatible with screen readers.
These adaptations actively reduce participation barriers, promoting a more inclusive and representative data collection process, aligning with the goals of social surveys in Official Statistics to minimise total survey error. By reducing barriers, the GPPS strengthens its ability to provide robust, reliable data reflecting all patient experiences, including those often marginalised. As a result, the GPPS is widely used to understand inequalities in healthcare experience.
This presentation will discuss the practical implications of these accessibility measures and their contribution to a more equitable and representative understanding of NHS patient experiences. It will also explore the potential of these formats to inform best practices for other social surveys, promoting more inclusive data collection methodologies.
Mrs Beverley Bates (National Centre for Social Research) - Presenting Author
Mrs Sarah Morris (National Centre for Social Research)
Adult Psychiatric Morbidity Survey (APMS) is a flagship mental health survey in England. Our client on the survey, NHS England, wished to add an ethnic minority boost to the survey to improve the information that is available on the mental health of specific ethnic groups. We worked with a communications agency focused on minority communities to create an inclusive and culturally sensitive communications and fieldwork strategy. In the presentation, we will give an overview of what worked about this approach in particular and what were wider issues that eventually resulted in the ethnic minority boost not meeting its aims. We will also explore whether an even more inclusive approach would be feasible in the context of a large survey and ask the wider question of whether boosts of any small group are still viable in face-to-face surveys.
Mr Tony Kelleher (Central Statistics Office) - Presenting Author
Ms Fiona O'Riordan (Central Statistics Office)
11th Conference of the European Survey Research Association (ESRA)
From Sampling to Collection.
Strategies that make surveys more inclusive and equitable in Ireland
Abstract
Social data is key to a properly functioning society. Our age, our sex, our educational attainment, are we working? do we have a warm coat? have we experienced discrimination? is data that informs government, researchers, and the citizen about our lives.
The task of collecting social data has changed considerably over the last 5-10 years though. Falling response rates have posed real challenges to ensuring quality representative data collections. The CSO has been active in moving to a multi-mode environment, (similar to other NSOs) and adapting survey design to suit the requirements of the respondent. Providing choices for the respondent is part of this work but to do this, it is important to understand their requirements.
This work spans across several processes from sampling to collection. CSO continually reviews sample, design, availability of administrative data to reduce burden and collection modes.
Some of our initiatives are below and we continue to learn about our respondents from our respondents.
• The CSO have reviewed our sample design to ensure they are efficient, and the sample is as close to the target as possible.
• We are mindful of literacy challenges in questionnaire design. We use simple language when formulating questions and designing questionnaires.
• We engage with advocacy groups early and often to ensure that everyone is being represented and all voices are heard.
• The CSO knows how important the concept of Trust is to our respondents and we continue to treat this agreement between us and them as an important agreement.
There is no one solution to ensuring everyone participates in these important surveys, but it important to continually review and modify to ensure the collection of
Dr Alyce Raybould (University College London) - Presenting Author
Professor Lisa Calderwood (University College London)
Professor Alissa Goodman (University College London)
Professor Pasco Fearon (University College London and University of Cambridge)
Dr Erica Wong (University College London)
Ms Karen Dennison (University College London)
Birth cohort studies are vital for understanding the development of successive generations of children, though there is increasing recognition that often those families who are of most interest from a research and policy perspective are less likely to be recruited and retained in these studies. Therefore, designing an inclusive survey that can effectively recruit and retain hard-to-reach populations is a key challenge.
The Early Life Cohort Feasibility Study (ELC-FS) aimed to test the feasibility of conducting a UK-birth cohort study, collecting information about several thousand babies and their families between 8-12 months old in 2023-24. The ELC-FS has a very strong focus on inclusivity and was designed to maximise representation of ‘less often heard’ groups. It includes sample boosts of babies from disadvantaged and ethnic minority families, as well as Scotland, Wales and Northern Ireland, and direct recruitment of fathers including those living in their own households. The study also made concerted efforts to build accessibility into the study’s design.
The study involved 30-60 minute interviews with mothers and fathers. We also collected record linkage consents and saliva samples for DNA extraction. Interviews were carried out primarily face-to-face, with web, phone and video interviewing also used. We also tested the effectiveness of targeted differential incentives and different conditional incentives.
This paper will give an overview of the inclusive and accessible design of ELC-FS. We will present some early findings relating to representativity, response rates, and the experimental components of the study: we achieved a high proportion of lower-income families in the sample, response rates among ethnic minority groups tended to be higher than others, and we found differences in consent rates to data linkages and saliva provision among lower income and ethnic minority families.
Dr Alice Toms (Office for National Statistics) - Presenting Author
The UK Government Statistical Service Harmonisation team within the Office for National Statistics (ONS), is currently working on updating the disability and impairment harmonised questions (‘standards’). Disability is a complex topic, and so significant work has been undertaken to understand the problem space and gather a broad range of user needs and requirements. This talk will detail our qualitative approach to discovery, including focus groups, workshops, and cognitive testing of the existing and alternate versions of the disability and impairment standards.
The work is part of a wider suite of research within the ONS Future Population and Migration Statistics programme. The disability workstream focuses on several questions, including the long-lasting health conditions and illness standard, the activity restriction standard, and the impairment standard. These standards are designed to align with the Equality Act 2010 or the Disability Discrimination Act (1995) for Northern Ireland: ensuring comprehensive data collection that reflects the diverse experiences of individuals who are disabled. In particular, they capture the core disabled population according to UK legislation. The update aims to ensure increased inclusivity of the wider disabled population, whilst ensuring coherence, comparability and consistency of our data and statistics.
A recent review of the standards by the ONS Harmonisation team showed that user requirements have changed significantly over the last decade. There is an increasing argument for viewing disability in line with the social model of disability, rather than according to legislation. There are also data gaps where complementary surveys that captured additional progressive health conditions and other non-core elements of the disability standard have either stopped running or changed over time. Updating the standards is pivotal in working towards standardizing the collection of data to enhance the accuracy and comparability of statistics across the UK, as well as improving the inclusivity of our data.
Dr Tansy Arthur (non-member)
Ms Hayley Forrest (non-member) - Presenting Author
Project Overview
This project is about inclusion for a hard-to-reach population group. The Office for Veterans’ Affairs (OVA) commissioned the Office for National Statistics (ONS) to carry out the Veterans’ Survey to support the aims of their Veterans’ Strategy Action Plan, to make the UK the best place for veterans to live by 2028.
The research aim was to understand the unique experiences of the UK veteran population and their families living in the UK to inform future government policy.
We included a diverse group of stakeholders. Having broadened the scope from England to the UK, we worked with the Welsh Government, Scottish Government and Northern Ireland Veterans’ Support Office. Charities and Veterans’ Organisations supported the project.
Sample
A self-select sample was used for the veterans’ survey as a cost effective and inclusive method. Veterans are hard to reach, and a random household sample would include many households with no veterans, incurring additional costs. Data Linkage with the 2021Census would enable us to compare responses to population estimates.
Survey Development and Operation
The accessible survey was online first, with paper for those with no internet access. Questions were designed and tested collaboratively with veterans and other stakeholders. The collection ran from Armistice Day (11 November 2022) for 12 weeks. The survey promotion was through veterans’ charities and organisations. The overall response rate was over 24,000 and well above the target of 13,000.
Data Linkage
The processed data was linked to the 2021Census to measure representativeness. This showed a match of 98.4%, with differences in 2 regions (one higher, one lower), and fewer of the oldest veterans participating.
This paper will discuss the self-select survey method enabled by UK veterans charities and organisations, the data linkage to the 2021Census, and the published results.
Dr Marieke Haan (University of Groningen) - Presenting Author
Dr Yfke Ongena (University of Groningen)
Dr Rita Smaniotto (University of Groningen)
Ms Demi Stadens (Municipality of Ermelo)
The decline in survey response rates across all survey modes poses critical challenges for survey research. Traditionally, insights into non-participation have been drawn from paradata and brief non-response surveys administered post-refusal.
To deepen our understanding of survey (non)-participation, we conducted qualitative semi-structured interviews with both respondents and non-respondents. Guided by established survey theories and practical insights, we designed an interview guide to explore experiences with and reasons for (non)-participation. The guide included questions on motivation, survey design, survey topics, communication, and privacy (Dillman, 2020), as well as vignettes and photos to encourage conversation. Recruitment proved challenging (see Haan et al. 2024), but through personal professional networks and snowball sampling, we recruited 24 participants between September 2023 and July 2024 — 12 respondents and 12 non-respondents. Interviews were followed by an assessment of non-respondent types: occasional non-respondents and "hard" non-respondents being very resistant to participate.
Thematic analysis of the transcribed data revealed that negative past survey experiences have a lasting impact, reducing willingness to participate. While positive experiences, such as gaining insights or reflecting on personal views, increases engagement. We will present these findings, discuss improvements for the survey process and discuss, with an eye on the results, how surveys can potentially be made more inclusive.
References:
Dillman, D. (2020). Towards survey response rate theories that no longer pass each other like strangers in the night. In P.S. Brenner, Understanding Survey Methodology. Sociological Theory and Applications (pp. 15-44). Springer.
Haan, M., Toepoel, V., Ongena, Y.P., & Janssen, B. (2024). Recruiting non-respondents for a conversation about reasons for non-response: A description and evaluation. Survey Practice 17 (March).
Mr Chris Martin (Ipsos) - Presenting Author
Ms Rachel Ormston (Ipsos)
This paper will analyse historic outcome data of the three major surveys in Scotland to examine whether hard to reach groups have become harder-to-reach, the potential impact on non-response bias, and the implications for future survey design. Major face-to-face random preselected surveys in Scotland, like their counterparts in the rest of Europe, play a crucial role in informing public policy. Face-to-face designs have been the preferred design partly because they achieve higher response rates and better representativeness than other modes. However, falling response rates and increasing costs have meant greater consideration has been given to alternative designs.
While overall trends in response rates show a clear decline over time, what is less clear is the impact that this has had on nonresponse bias. Although response rates have typically been used as a key indicator of survey quality and a measure of potential bias, the relationship between response rates and nonresponse bias is complex and a high response rates do not necessarily equate to low nonresponse bias (nor low response rates to high nonresponse bias.)
Particular types of groups have traditionally been under-represented in face-to-face surveys. These include younger adults, people from minority ethnic backgrounds, those with lower levels of education, those with lower literacy levels, and those living in flats. However, it is less clear if, relative to the overall population, those groups that have been traditionally hard-to-reach are becoming harder-to-reach over time, or whether they are sharing equally in the overall decline in response rates. The findings will contribute to boarder discussions of the future of survey methodologies and value for money on traditional face-to-face approaches.
Dr Wendy van Rijswijk (Department for Education) - Presenting Author
Mr Joas Flynn (Department for Education)
Ms Laura Hewitson (Department for Education)
Ms Samantha Spencer (National Centre for Social Research)
Ms Line Knudsen (National Centre of Social Research)
Over eighteen per cent of children and young people in England have identified Special Educational Needs or Disabilities (SEND). A targeted longitudinal study would help us understand their experiences and subsequent educational and adult outcomes.
We conducted a feasibility phase of the SEND Futures longitudinal study to:
1. assess whether representative participation and adequate levels of engagement could be achieved
2. identify design and implementation challenges to providing an inclusive survey for a population with heterogenous needs
This presentation will outline findings on engagement and attrition, drawing on both qualitative and quantitative data from parents/carers, young people, and interviewers.
The discovery phase involved two waves of fieldwork with young people with additional needs and their parent/carer. Wave 1 used face-to-face surveys for underrepresented groups and trialled an online survey with the wider SEND population. Wave 2 explored a web-first, mixed-mode design, which additionally offered telephone interviews to test whether overall engagement would increase and to provide an alternative means of participation to increase accessibility.
We will explore how types of additional needs were associated with participation across different modes, and why some young people were unable to participate. Furthermore, we will present new findings on perceived question comprehension and the level of support provided to young people to complete their surveys, and how this varied by needs.
We will reflect on the implications of these findings for understanding accessibility and how we can and should support young people to take part. We will set out potential challenges that arise from this, such as how the presence of and/or support from others might affect comparability and validity of the collected data. Finally, we will discuss how to balance inclusivity and accessibility with practical decisions (e.g., resources and timelines) in the design.
Dr David Ryan (Bradford Institute for Health Research, Bradford Teaching Hospitals NHS Foundation Trust, BD9 6RJ, Bradford, UK ) - Presenting Author
Dr John Pickavance (Bradford Institute for Health Research, Bradford Teaching Hospitals NHS Foundation Trust, BD9 6RJ, Bradford, UK )
Dr Katy A. Shire (Bradford Institute for Health Research, Bradford Teaching Hospitals NHS Foundation Trust, BD9 6RJ, Bradford, UK )
Professor Rosemary R.C. McEachan (Bradford Institute for Health Research, Bradford Teaching Hospitals NHS Foundation Trust, BD9 6RJ, Bradford, UK )
Background:
Since 2007, the Born in Bradford (BiB) longitudinal birth cohort study has been following the health and wellbeing of over 13,500 people across Bradford, one of the most ethnically and socio-economically diverse cities in the UK. The cohort is now experiencing adolescence, and since 2022, BiB Age of Wonder (AoW) has been capturing the journey through adolescence and adulthood for the BiB cohort and their peers. With up to 30,000 adolescents involved over a seven-year period, AoW is one of the world’s largest studies of adolescence, and places large scale, representative survey data collection at its core.
Methods:
Survey data on health and wellbeing (including mental health, socioeconomics, and physical health) is collected annually from students in years 8, 9, and 10. Surveys are delivered in schools either online using computers or offline using tablets. All aspects of the AoW surveys are co-produced with young people, teachers, and parents to ensure alignment with schools’ and students’ changing needs and priorities.
Results:
Survey data was collected on over 9,500 adolescents in the academic year 2023/2024. Over 60% of respondents were from ethnic minority groups, while over 25% were eligible for free school meals (a commonly-used proxy measure for socioeconomic deprivation in the UK). Offering diverse survey delivery models to schools, minimising survey length and co-designing measures with young people enabled response rates of over 65%. Responsive dissemination efforts via interactive data dashboards helped to bolster study effectiveness and sustain participant engagement.
Conclusion:
Embracing co-production, partnering with young people and schools, and enacting responsive research practices has enabled AoW to collect large-scale, representative survey data on adolescents’ health and wellbeing across Bradford. In doing so, AoW provides a model for co-producing longitudinal survey research in secondary schools.
Mr Dalain Williams (Google) - Presenting Author
Mr Luke Remy (Google)
This presentation explores the crucial role of survey research in measuring and enhancing product inclusion, equity, and accessibility (IEA). While AI offers exciting possibilities for optimizing products and services, it also presents unique challenges in ensuring equitable outcomes for diverse user groups.
We argue that a holistic approach to IEA measurement requires moving beyond traditional metrics and embracing a multi-faceted framework that incorporates:
Quantitative data: Analyzing user demographics, behavior patterns, and satisfaction levels across diverse populations through surveys, A/B testing, and digital analytics.
Qualitative data: Gathering rich insights into user experiences, needs, and pain points through focus groups, interviews, and user feedback mechanisms.
Ethical considerations: Embedding principles of fairness, transparency, and accountability in the design, development, and deployment of AI-powered products.
This presentation will showcase innovative survey methodologies and AI-driven tools that can help researchers and practitioners:
Identify and mitigate bias: Uncover potential biases in algorithms and datasets that may lead to discriminatory outcomes.
Personalize user experiences: Tailor products and services to meet the unique needs of diverse user groups.
Measure the impact of IEA initiatives: Evaluate the effectiveness of interventions aimed at promoting inclusion, equity, and accessibility.
By integrating these approaches, we can ensure that AI technologies are developed and deployed in a way that benefits all members of society, leaving no one behind. This presentation aims to spark discussion and collaboration among researchers, developers, and policymakers to advance the field of inclusive AI and create a more equitable future.
Mr Todd Hughes (University of California Los Angeles) - Presenting Author
Dr Ninez Ponce (University of California Los Angeles)
Since its inception in 2001, a central objective of the California Health Interview Survey (CHIS) has been to reflect the very diverse population of the state of California, including sufficient representation of race and ethnic minorities, those with language needs other than English, persons with different levels of education and literacy, sexual and gender minorities, and other under-represented groups. The CHIS has employed a multifaceted set of strategies to address these challenges, including predictive modeling during address-based sampling, multi-mode data collection, supplemental sampling frames including prepaid cell phones, multiple strategies for recruitment and data collection in languages other than English, linguistic simplification review of survey questions and materials, oversampling of specific race and ethnic groups, and follow-on studies of understudied groups. In recognition of these efforts, the CHIS was chosen as the winner of the 2024 American Association of Public Opinion Research (AAPOR) Inclusive Voices Award, which recognizes important data sets, research, and survey methods that have improved the ability to study complex social phenomena related to understudied populations. This presentation will describe the various methods used in the CHIS to address inclusivity for the diverse population it studies, as well as some ideas to consider for future strategies.