All time references are in CEST
Survey research beyond the binary - Exploring the potentials, challenges and consequences of alternative measures for assigned sex, gender identity/expression and sexual orientation |
|
| Session Organisers | Professor Stephanie Steinmetz (University of Lausanne) Dr Verena Ortmanns (DIE Bonn) Dr Lisa de Vries (Bielefeld University) Dr Angelo Moretti (Utrecht University) Dr Katharina Meitinger (Utrecht University) Professor Mirjam Fischer (Humboldt-Universität zu Berlin) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Awareness of gender diversity beyond the heteronormative male-female binary has grown rapidly in Western society. As recognition of diverse gender identities increases, the question of how to reflect this in surveys arises. With 39 out of 47 Council of Europe member countries now having legal processes in place to recognize trans and gender nonconforming (TGNC) people, this is a timely and necessary conversation. Using more inclusive measures in surveys not only promotes greater accuracy but also ensures a more equitable representation and visibility of gender diversity. It allows individuals to answer questions and define themselves according to their identity.
In this context, survey providers and population registers/censuses have shown a growing interest in more accurately capturing gender diversity in their data collection efforts in recent years. This shift, however, presents challenges and potentials for survey design, particularly for questions related to sex (assigned at birth), gender (expression and identity) and sexual orientation. These changes may also affect respondents' responses to other survey questions. A key issue is balancing the need for valid and reliable survey measures with the goal of being inclusive of gender-diverse populations, such as trans and non-binary individuals. Traditional survey methods may not adequately reflect this diversity, but new approaches may also pose challenges, particularly in terms of whether they are interpreted consistently by the general population.
Against this background, the session welcomes papers that provide practitioners, survey designers, and researchers interested in assessing assigned gender, gender identity/expression, and sexual orientation with new tools and measurement strategies; address the benefits and risks of including gender diversity in surveys and official registries/censuses while ensuring the validity and reliability of these measures; and/or address cross-cultural challenges, such as those related to translation. We especially welcome contributions that provide insights and perspectives
Keywords: sex, gender identity, sexual orientation, measurement, gender minority people
Mr Todd Hughes (University of California Los Angeles) - Presenting Author
Dr Tara Becker (University of California Los Angeles)
Dr Ninez Ponce (University of California Los Angeles)
The measurement of transgender and gender expansive populations has become increasingly common in population surveys. The most common method used in the USA is the two-step gender identity sequence. Differences across surveys in the set of gender identity response options that are provided to respondents may affect the ways in which respondents report their gender identity. The California Health Interview Survey (CHIS) provides a unique opportunity for understanding how the response options provided to respondents shape the gender identification they report. CHIS introduced a two-step gender identity sequence to its annual adult survey in 2015, and its annual adolescent survey in 2019, allowing measurement of trends over time in gender identification in California and in the size of California’s transgender and gender expansive population. In 2023, CHIS modified this two-step sequence to introduce a new non-binary response category to the gender identity question, and adopt additional changes. As a result of these changes, we see a large shift in gender identification from the “Transgender” to the “Nonbinary” category, as well as a substantial reduction in the number of respondents who choose to write-in another term. However, despite these changes, estimates of the overall size of the transgender and gender expansive population remained consistent with existing trends within both the adult and adolescent populations. The results suggest that within the nonbinary population there are at least two different groups of respondents: those who will identify as “Transgender” when no other gender expansive response option is provided and those who prefer to write in a nonbinary gender identity. This complicates estimates of the size of the nonbinary population when no explicit nonbinary option is provided, but also suggests a significant proportion of the nonbinary population may not see themselves as part of the broader transgender community.
Dr Christopher Hansen (NORC at the University of Chicago) - Presenting Author
Dr Ipek Bilgen (NORC at the University of Chicago)
Dr David Dutwin (NORC at the University of Chicago)
Valid and reliable measurement of sexual orientation and gender identity (SOGI) remains of critical importance to surveys and survey panels. Survey organizations balance respondent experience (e.g., inclusivity, cognitive burden, etc.) with their clients’ need for representativeness and oversampling of sexual and gender minority (SGM) groups. SOGI measurement is an evolving area of survey science, with recent scholarly work offering new approaches to measuring lived experience outside traditional binary conceptions of sex, gender, and sexuality. Our presentation contributes to this research area by sharing results from SOGI experiments conducted during 2024 recruitment for AmeriSpeak®, a probability-based household panel operated by NORC at the University of Chicago.
Data for this study were collected in English and Spanish from April-December 2024 as part of a panel recruitment survey conducted by sequential web, phone, and in-person data collection modes. Approximately 3,000 newly recruited panelists were randomly assigned to 1 of 3 SOGI batteries that varied in terms of response options, including the use of open-ended text boxes, and 2- versus 3-step gender identity measurement. The first battery tested measures recommended by the U.S. federal government; the second tested measures with higher-specificity response options as well as an additional item for gender modality (i.e., cisgender/transgender); and the third tested AmeriSpeak’s existing measures for comparison purposes.
Our presentation provides results from these experiments, including a comparison of response distributions (e.g., prevalence estimates of SGM groups) and data quality indicators (e.g., item nonresponse, breakoffs) by battery. We will also present cross-cultural considerations (e.g., differences by language) as well as potential mode effects by web, phone, and in-person administration. Lastly, we will discuss the findings’ implications for future panel data collection and their contributions to the SOGI measurement literature more broadly.
Mrs Johanna Dau (Institute for Advanced Studies (IHS, Vienna)) - Presenting Author
For over 25 years, researchers grounded in queer theory have criticized the gap between methods used to measure gender and theoretical understandings of gender. As trans, intersex, and non-binary (TIN) individuals gain visibility, scholars across disciplines have worked to improve gender measurement, emphasizing the need for inclusivity, validity, and research-practicality.
This presentation examines the challenges of measuring gender in large-scale, cross-national surveys using the EUROSTUDENT project – a study harmonizing student surveys from 27 European countries – as a case study. It addresses barriers to comparability, alignment with administrative datasets, and designing inclusive questions that capture gender diversity while ensuring robust analyses.
Drawing on international literature and my experience with gender measurement in student surveys, I present three approaches: an old-fashioned two-step design used in EUROSTUDENT rounds 7 and 8*, an improved three-step design implemented in the Austrian National Student Social Survey 2023, and a streamlined two-step approach developed for EUROSTUDENT 9, taking place in spring 2025. Empirical data and metadata, such as response times and dropout rates, enrich the discussion, highlighting trade-offs between inclusivity and operational feasibility.
The analysis explores balancing inclusivity with the demands of cross-national, large-scale surveys. It focuses on question formats, response options, and weighting processes, aligning with queer-feminist theory while maintaining compatibility with official statistics and minimizing respondent burden. Framed within Critical Quantitative Research, this work addresses the tension between deconstructionist theories and quantitative methodologies. It also proposes strategies for analysing small TIN populations in large datasets, offering actionable insights for capturing gender diversity in large-scale studies.
*EUROSTUDENT 7 (2019; unweighted N≈221,800; 17 countries in SUF); EUROSTUDENT 8 (2022; N≈194,800; 18 countries in SUF). Austrian Student Social Survey 2023 (N≈43,400) as part of EUROSTUDENT 8.
Professor Amy Alexander (University of Gothenburg)
Professor Susan Banducci (University of Birmingham) - Presenting Author
Professor Hilde Coffe (University of Bath)
Professor Jessica Fortin-Rittberger (University of Salzburg)
Professor Marta Fraile Maldonado (CSIC)
This paper introduces a new cross-national measurement approach to capturing non-binary measures of gender identity, drawing on data from the ESS Round 11 module, “Gender in Contemporary Europe: Rethinking Equality and the Backlash.” By expanding beyond the conventional reliance on respondent sex, our study integrates self-assessments of masculinity/femininity and trait-based indicators in a single survey instrument. This innovation addresses a key gap in cross-national research: the need for measures that clarify how gender identity varies among individuals identifying as men or women—and how these nuanced variations relate to broader social and political dynamics across diverse cultural contexts. We operationalize gender identity in three ways. First, respondents provide overall self-assessments of their own masculinity and femininity, allowing them to locate themselves on separate scales rather than being forced into a binary choice. Drawing on past research (Nilsson & Holmberg 2006; Wängnerud et al. 2019), these single-item scales capture respondents’ subjective views of their gender identities without imposing a binary definition. Second, following the Bem Sex-Role Inventory, respondents rate four traits commonly characterized as either masculine (take risks, being a leader) or feminine (compassionate, sensitive to others). Third, we capture self-reported gender (“what best describe you”), the first time self-reported gender has been asked in the ESS. Prior studies have largely used only one of these measures and often in a single-country setting, hindering opportunities for meaningful cross-national comparisons.
We detail the development of these items for the ESS, analyse the measurement properties and examine how masculinity/femininity and traits are related to one another and self-reported gender identity and binary sex. We also report how these relationships vary across cultural contexts. Overall, we demonstrate that these measures provide a unique opportunity to refine our understanding of
Dr Katharina Meitinger (Utrecht University) - Presenting Author
Dr Angelo Moretti (Utrecht University)
For decades, large-scale social science surveys measured gender as a binary question (Men/Woman or male/female). Since this question format often conflates sex and gender and does not account for the complexity of different gender identities, different more inclusive question formats have been proposed and adopted in large-scale surveys, such as adding an open-ended “Other” option or different two-step procedures. However, survey methodological research evaluating the different approaches were predominantly single-country studies. Less is known which version is preferred by the respondents and which version performs best in a cross-national survey.
As part of a NWO funded research project on cross-national measures of attitudes towards LGBT, we conducted a web probing study in six countries (France, Germany, Italy, the Netherlands, Poland, USA) in 2024 and presented respondents with three gender identity questions: 1) One question version with the answer options “man”, “woman”, “other, please type in”, 2) two-step version first asking for the sex recorded at birth and then whether the respondent describes him/herself as male, female, or transgender, 3) two-step version asking first whether the respondent was born male or female and then whether he/she describes him/herself as a man, woman, or in some other way. After responding to all three versions, the respondents were asked which version they preferred and why.
In this presentation, we evaluate whether the three different formats arrive at similar response distributions regarding cis- and non-cis respondents, discuss which countries and subgroups prefer which version and show the reasons for these preferences.
Professor Stephanie Steinmetz (University of Lausanne) - Presenting Author
Dr Verena Ortmanns (DIE Bonn)
The recognition and inclusion of non-binary gender expressions in survey methodology are gaining momentum, reflecting a shift toward capturing gender diversity. Despite this progress, many researchers continue to rely on sex (assigned at birth) as a proxy for gender, even when gender expressions are explicitly measured. However, dichotomous measures have been shown to be inadequate for capturing the complexity of gender. In contrast, finer-grained, continuous approaches provide a more nuanced and accurate understanding of gender disparities. Recent studies have particularly highlighted the benefits of disaggregating gender expressions into two independent dimensions - masculinity and femininity. This approach allows respondents to more precisely articulate their gender expression. Previous research, for instance, underscores the variation within gender expressions, even among cisgender respondents. While femininity and masculinity often align with identifying as a woman or man, significant within-group variation exists. Adherence to traditional gender norms, such as masculine men and feminine women, is linked to higher marriage likelihood and a traditional family model, while gender nonconformity correlates with distinct health outcomes.
Against this backdrop, this paper explores the implications of using continuous, bi-dimensional scales for measuring masculinity and femininity, rather than dichotomous categories (e.g., men vs. women), to capture gender differences in attitudes toward gender roles and traditionalism. Using data from the German GESIS Panel and the Swiss MOSAiCH study, participants reported their sex assigned at birth, their gender identity using a non-dichotomous measure, and their degree of gender expression using bi-dimensional scales. The analysis also considers demographic characteristics and attitudes. The study aims to strengthen the case for finer-grained gender measures in (population-based) surveys by demonstrating how continuous measures better capture the diversity of gender experiences. With that, we seek to advance the methodological toolkit for studying gender differences and inequalities.
Dr Christina Bornatici (FORS - Swiss Centre of Expertise in the Social Sciences)
Mr Max Felder (FORS - Swiss Centre of Expertise in the Social Sciences) - Presenting Author
Dr Lavinia Gianettoni (University of Lausanne)
Mrs Roxane Mordasini (f EFORS - Swiss Centre oxpertise in the Social Sciences)
Professor Stephanie Steinmetz (University of Lausanne)
Gender differences are among the most scrutinized in social sciences analyses. Quantitative surveys commonly employ a single binary question to categorize respondents and study gender differences. However, this practice falls short in distinguishing between assigned sex and gender identity, resulting in mismeasurement and risking the essentialization of gender differences. This poses challenges, especially as society increasingly recognizes that assigned sex is distinct from gender identity, and these two aspects may not align within individuals.
Issues of gender identity inclusivity have become a major societal concern. It is increasingly recognized that biological sex is not dichotomic and may change over time, while gender identities are diverse, fluid, and expressed in various ways. Additionally, sexual orientation constitutes another layer of self-definition. Simply measuring a male/female dichotomy cannot capture this diversity, calling for more inclusive and comprehensive measures in quantitative surveys.
Our goal is to enhance inclusivity and precision in social science surveys by effectively measuring assigned sex, gender identity, gender expression, and sexual orientation. This involves optimizing survey questions to provide a more comprehensive understanding. Using inclusive measures fosters more representative datasets, allowing individuals to define themselves according to their felt identity. Beyond fostering inclusivity, measuring these concepts with finer categories enables more refined analyses of inequalities.
Our paper examines current practices in Swiss and international social sciences surveys and present propositions for advancing gender measurement methodologies, discussing both challenges and opportunities in adopting more inclusive survey approaches.
Dr Verena Ortmanns (German Institute for Adult Education - Leibniz Centre for Lifelong Learning) - Presenting Author
Dr Katharina Meitinger (Utrecht University)
Dr Angelo Moretti (Utrecht University)
Dr Ranjit K. Singh ( GESIS - Leibniz Institute for the Social Sciences)
Gender is a multifaceted concept encompassing identity, expression, subjective experience, external perceptions, and performative behaviour, among other aspects. Despite its complexity, most surveys approach gender with a simplistic question such as, “Are you…?”, failing to specify which aspect of gender they aim to measure. Furthermore, response options are often restricted to binary categories (male and female), overlooking the nuanced and diverse nature of gender identities and concepts.
Previous research in Germany has shown that respondents can distinguish between different dimensions of gender and provide detailed self-assessments using continuous scales indicating the degree of being a man or woman or of femininity and masculinity. Building on this research, in the present study we implemented the full Traditional Masculinity-Femininity Scale (TMF), a six-item measure of individual’s gender-role self-concept. This scale captures a higher-order masculinity-femininity construct that transcends specific attributes, such as traits, interests, physical characteristics, or attitudes. However, this scale does not exclude variation on masculinity-femininity within a person depending on different social, temporal, or regional contexts. Moreover, this study expands previous research by conducting web-based surveys using non-probability samples in six countries: France, Germany, the Netherlands, Poland, Spain, and the United States. The cross-national design enables us to analyze differences in response behaviors across cultural contexts.
We hope to demonstrate the complexity of gender concepts when survey response options allow for greater gradation. Results are presented for each country, examining whether respondents utilize the full range of the scale or cluster at its extremes. Furthermore, we explore the interplay between different layers of gender concepts and assess the sensitivity of responses to these nuanced dimensions.