ESRA 2025 Preliminary Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

All time references are in CEST

Conventional and Innovative Sampling Methods for Hard-to-Survey Populations
Session Organiser	Dr Melike Saraç (Hacettepe University Institute of Population Studies)
Time	Tuesday 15 July, 15:30 - 17:00
Room	Ruppert D - 0.24

Hard-to-survey populations typically refer to sub-groups of the total population that are difficult to define, cover, select, interview, and adjust for post-survey procedures in surveys. It is important to include these sub-groups in surveys so that policy makers can develop evidence-based policies in many areas (e.g., health, nutirition, disability, education, labor) for these groups relying on survey or registration data. Tourangeau (2014) discusses the terminology of hard-to-survey populations in the survey context. He elaborates on the distinction between hard-to-sample, hard-to-identify, hard-to-find and hard-to-interview in detail.

This session aims to discuss conventional and innovative sampling and data collection methods, in particular to cover hard-to-survey populations targeted in social surveys. The session welcomes studies dealing with methods for the sample frame, sample selection (probabilistic vs non-probabilistics), data collection, fieldwork supervision, and post-survey adjustments including weighting and calibration. The pros and cons of new sampling ideas in the digital age will be discussed in addition to traditional methods. Comparative or experimental studies as well as studies based on mathematical calculations are therefore also to be welcomed. Researchers from different countries would contribute by sharing their survey experience of sampling and collecting data from hard-to-survey populations.

Keywords: sampling, hard-to-survey populations, coverage, weighting

Papers

Hard-to-reach Groups in a Survey Design: Experiences and Findings from Statistics Austria‘s Migration Survey on Incentives and Translations

Dr Jonas Kolb (Statistics Austria) - Presenting Author

Migration and integration require more than just data on population development or immigration and emigration; they are also an expression of subjective assessments and perspectives. The migration survey is a sample survey carried out annually by Statistics Austria. The focus is on various dimensions of migration and integration.
A net sample of 4,300 people is targeted annually for the migration survey. This includes people in Austria without a migration background as well as immigrants from the selected countries of birth Bosnia and Herzegovina, Turkey, Serbia, Afghanistan, Iran, the Russian Federation, Somalia, Syria, Romania and Ukraine. The participants are drawn from the Central Register of Residents (ZMR) according to a multi-stratified procedure, whereby in addition to the country of birth, a restriction is also made to persons aged 16 and over. In the case of immigrants, they must have been resident in Austria for at least twelve months.
The presentation introduces the migration survey and provides insights into the survey design. Particular attention is paid to how immigrants are reached as hard-to-reach groups. First, we will discuss how incentives are used in the migration survey. As a post-incentive, respondents can choose between a EUR 10 voucher and a donation worth EUR 10 for a nature conservation project in Austria. The presentation will discuss the distribution of socio-economic backgrounds in the choice of incentives among immigrants.
The presentation also explains how different languages are dealt with in the migration survey. The survey is conducted as an online survey and can be answered in German, Arabic, Bosnian, English, Farsi, Romanian, Russian, Serbian, Somali, Turkish and Ukrainian. The challenges of implementing a multilingual questionnaire are examined, as well as findings regarding the use of foreign languages by respondents.

Incentivizing Firms in Surveys - Evidence from a CATI Experiment

Mr Maik Sattelmaier (University of Mannheim) - Presenting Author
Professor Davud Rostam-Afschar (University of Mannheim)
Mr Fabian Eble (University of Mannheim)

Do monetary incentives change response behaviour in Computer Assisted Telephone Interviewing (CATI) surveys? We conduct a CATI survey experiment on incentives with a hard-to-reach population subgroup - CEOs and firm decision-makers from mainly small and medium-sized firms active in Germany. The sampling frame consists of firms with revenues above 2 million and more than 10 employees. For this economically significant subgroup, we elicit fundamental preferences such as risk aversion, time preferences, the resolution of uncertainty, and the elasticity of intertemporal substitution. First, we examine whether response behaviour differs in directly incentivised questions for different incentive levels. We investigate whether there are significant differences in the centre of the distribution and whether extreme responses increase in the non-incentivised subsample. In the second step, we investigate whether incentivisation leads to spillover effects to the non-incentivised questions, such as revenues, item-nonresponse, the willingness to answer a question on tax burden, which is an example for a sensitive question, and the willingness to provide their personal email address for a follow-up contact. The questions are framed in the context of actual business decisions that apply homogeneously to all type of firms - tax payments and interaction with fiscal authorities. We are therefore able to elicit realistic preferences from the entrepreneurs. Our results provide insights into the use of monetary incentives in CATI surveys, preference elicitation for a hard-to-reach population subgroup and cost-efficient implementation.

Using the Directed Seed Method for Respondent-Driven Sampling to Survey Venezuelan Refugees and Migrants in Colombia

Professor Katherine McLaughlin (Department of Statistics, Oregon State University) - Presenting Author

Respondent-driven sampling (RDS) has been adopted worldwide as an important method for sampling vulnerable and high-risk populations to enable vital decisions about resource allocation and program planning and implementation. However, RDS relies on many assumptions about the population and sample dynamics, and continued innovation is needed to ensure that RDS surveys are implemented correctly to meet these assumptions. In this paper, we introduce the directed seed method as a modification to RDS where seeds work with an interviewer to enumerate their potential recruits across a number of important characteristics using a diversity recruitment grid and are then instructed on which of these people they should recruit. The directed seed method shows promise for enhancing the recruitment of additional diverse individuals and overcoming potential bottlenecks in the population network by concentrating bias in the seeds and their recruits and then allowing the RDS sample to proceed without interference. We found that directing seeds to recruit across identified bottlenecks can be helpful to reach a more diverse group of population members sooner and to try and ensure that all important subgroups in a population are included in the final sample. We provide an example from surveys conducted among Venezuelan refugees and migrants in Colombia in 2020 to introduce the steps needed to implement the directed seed method. We assess the method using a variety of existing and novel diagnostic tools, including visual diagnostic techniques like recruitment chain, bottleneck, and all points plots and metric-based diagnostic techniques like recruitment homophily, expected gain in wave 1 diverse participants, and standard deviation of seed-based characteristic estimates based on a threshold. Finally, we discuss best practices for implementation.

UK Veterans Survey using self-select and data linkage method

Dr Tansy Arthur (Non-member (at the moment)) - Presenting Author

Project Overview
The Office for Veterans’ Affairs (OVA) commissioned the Office for National Statistics (ONS) to carry out the Veterans’ Survey to support the aims of their veterans’ Strategy Action Plan, which is to make the UK the best place for veterans to live by 2028. 
Stakeholders on the project included the Welsh Government, Scottish Government and Northern Ireland Veterans’ Support Office. Charities and Veterans’ Organisations such as COBSEO: The Confederation of Service Charities supported the project.
Background Rationale
The UK Veterans’ Survey was the first project of its kind in the ONS. The research aim was to understand the unique experiences of the UK veteran population and their families living in the UK to help inform future policy created by the government.
Sample
A self-select, respondent driven sample was used for the veterans’ survey. Veterans are hard to reach, and a random household sample would risk targeting households that did not have veterans, incurring additional costs, and low response due to missing veterans’ households.
Survey Development and Operation
The survey was online first, with paper for those with no internet access. Questions were designed and tested collaboratively. The collection ran from Armistice Day (11 November 2022) for 12 weeks. The survey promotion was through veterans’ charities and organisations. The overall response rate was over 24,000 and well above the target of 13,000.
Data Linkage
Once the data was processed it was linked to the 2021 Census to measure representativeness. This showed a match of 98.4%, with differences in 2 regions (one higher, one lower), and fewer of the oldest veterans participating.
This paper will discuss the self-select survey operation enabled by UK veterans charities and organisations, the data linkage to the 2021 Census, and the published results.

Surveying the Defence Team: Challenges and Practical Solutions

Dr Zhigang Wang (Department of National Defence) - Presenting Author

Reliable and accurate survey results about the military population and its subpopulations are required for evidence-based decision making in military organizations. To ensure the quality of survey data, design-based probability sampling methods and statistical inferences for finite populations are used in military survey research. Such standard sampling and analytical methods help survey researchers achieve the required precision of survey estimates for the total military population and subpopulations whose members can be accurately identified in the sampling frame (e.g., occupation groups). However, members of some military subpopulations (e.g., Indigenous people, members of visible minorities, persons with disabilities) are often imperfectly identified in the sampling frames, and some very small subpopulations (e.g., gender diverse) are impossible to identify in advance. Such constraints, in conjunction with non-responses, can result in insufficient and biased samples of these subpopulations and a biased sample of the total military population. These challenges threaten the quality of survey results and lead to inaccurate and unreliable empirical evidence.

This study shows how we overcome these challenges and ensure the scientific rigor of survey results, including our methods of sampling design (e.g., sampling frame development and sampling strategies), weighting and weight adjustments (e.g., weight adjustments for subpopulations that are imperfectly identified and impossible to identify in advance), and estimation and results reporting (e.g., analytic methods and results reporting for subpopulations with insufficient sample sizes). Advantages and disadvantages of these methods are discussed, and future methodological improvement are recommended.

Can roofs show us the way? – Innovative approaches to sampling in forced displacement contexts

Mr Patrick Brock (WB/UNHCR Joint Data Centre)
Mr Andrej Kveder (UNHCR) - Presenting Author
Mrs Anqi Pan (WB/UNHCR Joint Data Centre)

Collecting socioeconomic data in forced displacement contexts presents multiple methodological challenges and sampling in particular. Those affected by forced displacement are often vulnerable and hard to reach for the purposes of household surveys, and sampling frames are rare and of low quality. Therefore, many surveys in these contexts use less-than-ideal convenience-based sampling. As intervention planning, programming and policymaking in response to rising global forced displacement intensifies, the need for high-quality and comparable socioeconomic data on forcibly displaced grows.
This research explores the use of open data to respond to these challenges. An innovative sampling approach will be compared to alternative list-based sampling approaches, and we discuss the implications for survey implementation in terms of cost, logistics, and statistical estimation. The proposed paper discusses how combining UNHCR publicly available data with satellite imagery and Google Open Buildings data made an integrated sampling approach for the UNHCR Forced Displacement Survey in South Sudan possible.
The second challenge discussed in this paper is the comparability of data collected on forcibly displaced persons with those from the communities where they live. The main challenge lies in finding a generalizable definition of host communities, for example, in the degree to which host nationals are affected by the presence of refugees, such that it can be operationalized into a concrete sampling approach. The paper demonstrates the example of gravity sampling using Euclidian distance between host community dwellings and refugee camp boundaries in combination with a gravity coefficient. The paper also uses sensitivity analysis to explore how this approach overlaps with alternatives, which aim to more directly incorporate the interaction between refugees and host communities.
In conclusion we will investigate biases as well as representativeness of the samples and will elaborate on how they can be addressed through different weighting strategies.