All time references are in CEST
Survey experiments with hard-to-survey populations: Solutions or new challenges? |
|
| Session Organisers | Mr Randy Stache (BAMF Research Centre) Dr Laura Peitz (BAMF Research Centre) Mr Armin Kuechler (Bielefeld University) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Populations such as the homeless, elderly, irregular migrants, LGBTQIA+ individuals, victims of violence, and stigmatized people (e.g., individuals living with HIV or disabilities) are frequently underrepresented in general population surveys because they are particularly hard to reach. To address this issue, methodologies like respondent-driven-sampling and social media targeting have been developed.
However, reaching such populations is just the first hurdle; successfully conducting surveys with these groups presents additional complexities: Researchers are often interested in precisely those sensitive topics that make the groups difficult to reach, leading to low trust in the survey process and a higher social desirability bias or non-response to certain questions. Additionally, marginalized individuals may encounter difficulties with traditional survey frameworks due to factors such as low education, language barriers, or illiteracy, making it challenging for them to appropriately respond to questions or given formats. Health issues may further undermine survey quality.
Survey experiments, including factorial surveys and conjoint analyses, are increasingly used to tackle these challenges. In surveys with conventional populations, these methods were able to reduce social desirability through indirect questioning and were comprehensible to the participants due to their holistic design. However, there is limited research on the effectiveness of these experiments when working with hard-to-survey populations. Do these methods enhance comprehension and response quality? Can they effectively minimize social desirability bias and non-response? How do they compare to traditional direct questioning techniques? How can survey experiments be adapted to reflect the specific contexts and experiences of specific hard-to-survey populations?
This session will explore these questions, examining benefits and limitations of survey experiments within the context of hard-to-survey populations. In doing so, we aim to advance methodological approaches and improve the quality and inclusivity of survey research for often-overlooked research groups.
Keywords: survey experiments, hard-to-reach, hard-to-survey, factorial surveys, conjoint analyses
Dr Carlotta Minnella (EUI) - Presenting Author
Professor Martin Rush (EUI)
Mr Siegfried Manschein (EUI)
This paper seeks to reflect on the challenges and lessons learned in the design and execution of a major choice experiment with migration governance elites in Africa and Europe. The survey was conducted between September 2024 and January 2025 and targeted a pooled sample of representatives from the European Union (EU Commission, Parliament, European External Action Service, European Asylum Agency, FRONTEX), the African Union; as well as officials from the International Organization for Migration, the United Nations High Commission for Refugees, and from a select group of European and African countries’ ministries and institutions (e.g. Foreign Affairs, Interior, Labor, National Observatory for Migration). The design included a multi-institution and multi-country conjoint experiment as well as a standard survey questionnaire and sought to investigate elites’ knowledge claims and the related cognitive frames about the determinants of migration decisions between Africa and Europe. The paper presents the results of the survey and analyzes their scope, value, and possible limitations in view of the theoretical goals of the study, which was aimed at the measurement of ideational variables in migration decision-making. Additionally, the paper details and reflects on design choices and the recruitment strategy adopted to access the required sample of elite representatives, while mitigating the possible adverse effects of small sample sizes. The paper draws crucial lessons concerning the surveying of government elites in specific national and regional contexts, as well as the usefulness of elite surveys as opposed to other methods of enquiry (e.g. interviews and textual analysis) when conducting research on cognitive processes in elite decision-making. The survey is part of the EU Horizon 2022 project DYNAMIG, “How migration decisions are made: diverse aspirations, trajectories and policy effects.”
Mr Armin Küchler (Bielefeld University) - Presenting Author
Dr Marvin Bürmann (Bielefeld University)
This contribution examines how refugees evaluate potential long-term settlement opportunities in countries of first asylum, particularly in contexts where they face adverse conditions such as limited legal rights, restricted access to employment and inadequate social services. While traditional survey items often fall short in capturing refugees' complex decision-making processes regarding resettlement, our choice-based conjoint experiment conducted reveals how different factors are weighted in settlement and migration decisions.
The experiment presents respondents with paired comparisons of hypothetical destination countries, varying across eight key characteristics: family presence, diaspora size, crime rates, language skills, attitudes towards refugees, political system, labor market access, and lifestyle familiarity. Following each choice, respondents evaluate their perceived opportunities for building a future in both presented countries, enabling analysis of both relative preferences and absolute settlement potential assessments.
This approach offers distinct advantages for surveying refugee populations: it reduces social desirability bias through hypothetical scenarios rather than direct questions about migration intentions; provides concrete comparative choices that are more effective than single-vignette evaluations; and reveals how refugees trade off different aspects of potential host countries. The particular strength of our study lies in targeting refugees who are eligible for resettlement procedures, ensuring our findings reflect the preferences of individuals facing actual resettlement decisions rather than hypothetical migration scenarios. This methodological focus on respondents with practical resettlement prospects yields more reliable insights compared to studying general migrant populations who may lack concrete migration intentions. These insights directly inform resettlement program development and advance scholarly understanding of refugee migration preferences under precarious conditions.
The study draws on preliminary data from an online survey conducted jointly by BAMF-FZ and the University of Bielefeld, focusing on living conditions of particularly vulnerable refugees.
Mr Randy Stache (BAMF Research Centre) - Presenting Author
Dr Laura Peitz (BAMF Research Centre)
Ms Anne-Kathrin Carwehl (BAMF Reesearch Centre)
Hidden, marginalized, and elusive populations - such as drug dealers, elderly and immobile individuals, sex workers, sexual minorities, or migrants - pose significant challenges for survey research. These groups are often among the most disadvantaged and socially excluded, making it essential for scholars, civil society, and policymakers to understand their living conditions. While methods like respondent-driven-sampling (RDS) are increasingly used to reach such groups, questions remain about whether these populations are inherently difficult to survey and how best to design research for them.
Drawing on a survey with rejected asylum seekers, we explored the suitability of survey experiments for investigating sensitive topics in an explicit hard-to-reach and hard-to-survey population. From June to December 2023, the BAMF Research Centre conducted a fully app-based RDS survey targeting rejected asylum applicants from anglophone West Africa. The app integrated all key RDS elements and ensured participant anonymity by not collecting personal data. The participants were contacted via intensive formative assessment and an additional random sample from the Central Register of Foreigners.
Focusing on im-/mobility aspirations, the survey included both direct questions and a factorial survey where participants evaluated hypothetical scenarios of rejected asylum seekers in varying living conditions. Respondents made recommendations on whether these individuals should stay in Germany, return to their home country, or move elsewhere.
This paper critically examines the methodological approach, evaluating the effectiveness of factorial surveys in addressing sensitive issues within hard-to-survey groups. By analyzing survey behavior through paradata, we provide insights and lessons to enhance future research with these populations.