All time references are in CEST
Surveying Diversity: Methodological Issues and Best Practices |
|
| Session Organisers | Dr Lisa de Vries (Bielefeld University) Mrs Zaza Zindel (German Centre for Integration and Migration Research; Bielefeld University) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Diversity is a critical topic across society, encompassing dimensions such as age, gender, ethnicity, and socioeconomic status. As societies develop more complex and heterogeneous, inclusive research that adequately captures this diversity is essential. The task of surveying a “super-diverse” society, in which individuals embody multiple intersecting identities and experiences, presents unique challenges for survey design, data collection, and analysis.
This complexity not only opens new directions for survey research but also introduces significant methodological challenges. Traditional survey instruments may struggle to capture the full range of social diversity, and classic sampling techniques may fail to reach specific subpopulations. Furthermore, data security and research ethics become increasingly complex in this context. Addressing these challenges requires innovative approaches to sampling, questionnaire design, and data integration. Advances in technology and new data sources offer promising opportunities to better reflect the diversity of modern societies in research.
This session invites researchers and professionals from survey institutes, census bureaus, and related fields to discuss recent developments, as well as both methodological and practical challenges of surveying increasingly diverse populations. Given the societal relevance and its relatively recent emergence in survey research, we are open to a wide range of contributions that capture diversity in survey research. We especially welcome contributions that provide insights and perspectives from different communities (e.g., LGBTQI+, ethnic or religious minorities, or people with disabilities).
Relevant topics include (but are not limited to):
• Measurement instruments that reflect the complexity of a diverse society (e.g., intersectionality, multidimensional scales, tailored question formats)
• Sampling strategies that capture the full range of societal diversity, including hard-to-reach or underrepresented groups
• Balancing the need for detailed demographic information with concerns about respondent burden, privacy, and research ethics
• Integrating diversity across all survey stages, from question development to data analysis (e.g., participatory research
Keywords: diversity, inclusion, survey research, social heterogeneity, intersectionality
Mr Leo Valentin Theissing (University of Lucerne) - Presenting Author
Dr Tamara Bosshardt (University of Lucerne)
After the legalization of same-sex marriage in Switzerland in 2021 and (limited) access to sperm donation for lesbian couples, the possibilities for LGBTQIA+ people in Switzerland to have biological children increased. How do LGBTQIA+ people become parents? Little is known about how LGBTQIA+ families navigate their daily lives. Existing Swiss LGBTQIA+ studies (Eisner, Hässler 2024, Gfs Bern 2024) cover family only marginally. The representative household study of the Federal Statistics Office (BfS 2024), on families and generations, seems to be underestimating the number of same-sex couples ( 2%) and erases the complexity of queer families.
In our mixed-method research project, we aim to fill this research gap first with an explorative online survey on LGBTIQA+ families in Switzerland. In the second phase, we will deepen our understanding with qualitative research (ethnography, interviews, collaborative/creative research methods). The survey will be conducted via Qualtrics, in the three main Swiss languages and English. We tailor questions specifically to capture a wide range of families, whether formed over romantic, (queer)platonic, or other relationships, detangling social, biological, genetic, and legal parenthood (Peukert et al 2020). In our survey, we will ask LGBTIQIA+ people of all ages living in Switzerland with (wishes for) children their modes of family formation, family constellations, parenting and caring arrangements, interactions with institutions and specific challenges, and needs.
To ensure that we stay as close to the lived realities as possible, we use participative elements. Throughout the whole project, we collaborate with a community sounding board with queer parents and LGBTQIA+ community organizations like Dachverband Regenbogenfamilien. Our sampling strategy focuses on diversifying the sample beyond people who are already closest to the community organizations or have higher education (Dahl 2023). In our contribution, we will present our participative approach, sampling strategy, and some preliminary results.
Ms İrem Karaçay (University of Mannheim) - Presenting Author
Recruiting migrant populations remains a persistent challenge for population surveys. Even more challenging is the inclusion of ethnic minorities within migrant groups, as traditional methods often fail to account for the diversity within national origins. Relying on nation-state constructs—such as assuming that all individuals from Turkey are ethnically Turk and Turkish is their mother tongue—prevents the development of truly representative samples and undermines the validity of migration research. This study highlights the potential of tailored survey invitations in addressing these gaps by sampling Kurdish- and Arabic-speaking migrants from Turkey, as well as their descendants, residing in Germany through Facebook and Instagram advertisements.
Advertisements were carefully designed to align with the linguistic and identity markers of the target population to foster trust and participation. Without offering monetary incentives, the campaign collected over 1,900 completed questionnaires within 30 days on a budget of €7,000 (data collection is ongoing). Among Kurdish participants, 30% of those who began the survey completed the 20-minute questionnaire; for Arabic participants, the completion rate was 20%. Moreover, respondents actively engaged in open-ended questions, providing rich qualitative data.
The campaign also drew responses from non-targeted groups who contested the existence of ethnic groups other than Turks in Turkey, revealing the politically charged context surrounding these populations. This outcome underscores the complexities of sampling in transnational and contested identity spaces and points to new interdisciplinary directions for political science and survey methodology.
While this form of non-probability sampling carries limitations—such as self-selection bias and challenges to generalizability—this study demonstrates the need for more inclusive and innovative sampling approaches to counter the systematic exclusions present in traditional survey methodologies. By addressing these gaps, it enriches the methodological tools available to study underrepresented populations.
Mrs Ulrike Schwabe (German Centre for Higher Education Research and Science Studies) - Presenting Author
Mrs Frederike Gerdes (German Centre for Higher Education Research and Science Studies)
Dr Julia Steinkühler (German Centre for Higher Education Research and Science Studies)
Mrs Theresa Schommer (German Centre for Higher Education Research and Science Studies)
Dr Nancy Kracke (German Centre for Higher Education Research and Science Studies)
In recent decades, the student population in many countries has become more diverse (Kerst et al., 2024; Middendorff 2015; Middendorff & Wolter, 2021). In order to capture this diversity in all its dimensions, survey-designers have to develop an awareness for these complexities (Gottburgsen & Jungbauer-Gans, 2024). This contribution uses the Student Survey in Germany as a case study to demonstrate how student diversity is assessed in a nationwide, representative self-administered web-survey (Beuße et al., 2022).
Based on a general multi-layered concept of diversity in higher education, we first outline the measurement instruments employed, such as methods capturing gender, sexual orientation or prior educational pathways. Second, we describe our sampling strategy and weighting procedures. Particular attention is given to the challenges associated with (a) the role of higher education institutions as key gatekeepers for field access, (ii) the inclusion of students with health impairments, and (iii) the use of official statistics for post-stratification. We close our contribution with current empirical findings for Germany.
Professor Randall K. Thomas (Ipsos Public Affairs) - Presenting Author
Ms Megan A. Hendrich (Ipsos Public Affairs)
Survey researchers commonly use short filter questions to screen for individuals so they can be asked more in-depth questions. For instance, participants may be asked a general question like ‘Do you have any disability or impairment?’ If yes, the researcher can drill down to the type and extent of disability or impairment. However, this simple question may not sufficiently identify all people who might be of interest, including people who may have corrections for their impairment (e.g., hearing aids, crutches). We report on two studies (n=5,923; n=4,592) where we randomly assigned participants to no filter or a filter question (randomly using either a Dichotomous Response Format (DRF – ‘Yes-No’) filter or a Multiple Response Format (MRF – ‘Select all’) filter). All were then asked for their degree of impairment (ranging from ‘No impairment’ to ‘Complete impairment’), regardless of their filter response. The second study repeated the same manipulations as in the first study, but we followed up with items to determine the severity of the impairment more precisely. Both studies found a much higher impairment rate (hearing, seeing, movement, cognitive abilities, and mental health) for the ‘degree’ of impairment question than when using only a filter question. Though we identified more people with impairment, we found that the average rating of severity of impairment was less for those who did not indicate impairment in the filter question. In addition, the second study found that many who had lower levels of severity used aids for their impairment that were associated with a lack of identification of impairment in the filter questions. We conclude that simpler filter questions can significantly affect both the prevalence and severity of impairment in a population of interest. For more accurate impairment assessment, we recommend using graded impairment measures without filter questions.
Dr Laura Wortmann (Medical Faculty OWL, University Bielefeld) - Presenting Author
Dr Lena Haarmann (Medical Faculty, University of Cologne)
Mr Dima Bilyarchyk (Every Health)
Community-based research (CBR) offers a powerful approach to engaging so called hard-to-reach populations, particularly within the LGBTQIA+ community. This presentation discusses a survey conducted by the medical start-up Every Health, aimed at identifying healthcare challenges faced by LGBTQIA+ patients in Germany. With a total of 1007 participants, the survey, conducted between February and August 2023, combined both quantitative and qualitative methods, including open-ended items to capture nuanced lived experiences.
The success of the survey’s response rate is rooted in its community-driven design. Questions were developed and refined in close collaboration with LGBTQIA+ community members, ensuring the survey addressed the real needs, experiences, and pain points of the target group within the German healthcare system.
Recruitment took place through targeted online ads, social media outreach, and partnerships with queer organizations, ensuring broad and inclusive participation across Germany. The sample size and results of the survey demonstrate that so called hard-to-reach populations are often excluded due to research methods that do not resonate with their lived experiences. By conducting research with rather than on these communities, Every Health was able to achieve a high response rate and generate rich, meaningful data on the healthcare needs of LGBTQIA+ individuals.
After the survey, Every Health invited academic researchers to scientifically analyze the data, ensuring robust evaluation. This case study highlights how community-driven research and approaches can address the target population more adequately, and create comprehensive, and impactful findings.
Our presentation will explore the strategies employed through every step of the surveys design, implementation, and analysis, and the lessons learned in strongly engaging LGBTQIA+ communities and non-academic researchers through CBR. We will discuss how these insights can inform future research, not only with so called hard-to-reach populations but also in broader contexts
Mr Tom Luckwell (Verian Group) - Presenting Author
How best to survey minority groups within and across workplaces?
How to identify them and to reach them responsibly and ethically?
How to create anonymity and the conditions to speak openly while within the workplace?
How to write a survey that is relevant and meaningful while intersectional?
These were some of the challenges faced during the implementation of the ‘Our Black Workforce Survey’ 2022-2023. Commissioned by the Home Office UK and now managed by the College of Policing, the survey sought to understand and track the lived experiences of Black and Black-heritage police officers and staff across 44 police services. Focusing on sampling, fieldwork and questionnaire design, this paper uses the study as an example to explore various challenges and solutions to surveying minority groups within and across larger workforces and institutions.
Focus is given to the importance of fostering collaboration and open communication across respondents, senior management, line managers, human resources, and diversity & inclusion and communications leads. Such cooperation ensured the practicalities of delivery were met, as well as keeping internal dialogues around the study unified, positive and committed.
Attention is given to the unique application of ‘survey leads’, a police service staff or officer assigned to assist with sampling and fieldwork. Survey leads became a voice for the survey. Survey leads provided solutions to data concerns, enabling personal data to remain at source, and mitigate against everyday issues such as workplace firewalls.
Tensions between intersectionality and confidentiality when sampling are also considered. Regarding insight, measuring intersectionality was shown to reveal significant differences in lived experience at the intersections between ethnicity, rank, officer and staff roles and within and between Black ethnic subgroups. Finally, this paper explores approaches to integrating diversity through survey design, focusing on open text questions in trackers as rich feedback loops for development.