Collecting, Editing, Publishing and Improving Data on Hard-to-Reach Populations in Surveys 1 |
|
| Session Organisers | Professor Stephanie Steinmetz (University of Amsterdam) Dr Oliver Lipps (FORS) Dr Brian Kleiner (FORS) Mrs Jennifer Allen (Robert Koch-Institut) Mr Johannes Lemcke (Robert Koch-Institut) |
| Time | Friday 19th July, 09:00 - 10:30 |
| Room | D11 |
When surveying the general population the preferences, aspirations, or needs of the public as a whole should be revealed and every individual’s position should have the chance to be represented equally. Underlying this aim lies the notion of an observed sample as a representative, unbiased, and sufficiently precise reflection of an underlying (unobserved) population of interest of a study. Besides decreasing response rates, population-based surveys are facing additional challenges. One of them deals with the important aspect of how to access hard-to-reach populations. Frequently hard-to-reach populations are for instance, immigrants, refugees or ethnic minorities, sexual minorities, the elderly, the disabled, homeless people, children or households living in poverty, nomads, etc. The literature shows that hard-to-reach populations are not accurately represented in general population surveys. There are, systematically, higher non-response rates in cross-sectional surveys due to fewer contacts and more refusals as well as higher attrition rates in longitudinal surveys. The underlying mechanisms for the underrepresentation of those populations are more nuanced; they might be related to the fact that those populations are hard-to-sample, hard-to-identify, hard-to-find or to-contact, hard-to-persuade to participate and/or hard-to-interview (Tourangeau 2014). However, the consequence of the underrepresentation is that the survey is biased and that it may be difficult to estimate reliable and unbiased parameters for those populations. Besides general population surveys, the demand for surveys addressing special groups of the population (such as sexual minorities, refugees, sex workers, physically handicapped people etc.) has increased. However, those surveys come with particular requirements when it comes to sampling- and survey-design.
In this session, we would like to discuss the current situation of hard-to-reach population research in various national and international contexts as well as practical requirements and problems. This session invites contributions showcasing research around the challenges, successes, innovations and best practices when collecting, editing, and publishing data on hard-to-reach / vulnerable populations in surveys. We suggest but do not limit the session to the following aspects:
• Data quality
• Sample frame and coverage issues
• Sampling procedures, problems of accessibility
• Locating, contacting, and recruiting hard-to reach populations
• Unit- and item-nonresponse
• Interview modes
• New technologies and techniques in data collection
• Question properties and response sets
• Interviewer-Respondent interaction and interviewer effect
• Role of memory in rendering certain groups less visible to research
Keywords: hard-to-reach populations, underrepresentation, minorities
Dr Simona Kuti (Institute for Migration and Ethnic Studies, Zagreb) - Presenting Author
Dr Margareta Gregurović (Institute for Migration and Ethnic Studies, Zagreb)
Professor Saša Božić (Department of Sociology, University of Zadar)
Surveying immigrant populations represents a growing field within the social sciences. However, it includes various methodological challenges and potential limitations, from study design and sampling to data collecting and analysing. In order to discern different challenges researchers face, we shall undertake a scoping study – an approach to reviewing the literature following a methodological framework described by Arksey and O’Malley (2005).
Our scoping study will include journal articles published within the SocIndex (EBSCO) bibliographic database. The database covers "the broad spectrum of sociological study", and contains a number of relevant journals from the field of migration studies. Since the database includes a substantial number of journal articles, we will limit our search to articles published in English in 2008 – 2018 period with full text access. The study will implement a five stage framework provided by Arksey and O’Malley (2005), namely identifying the research question, identifying relevant studies, selecting studies, charting the data and collating, summarizing and reporting the results.
At the time of ESRA conference abstract submission, the search of SocIndex database for journal articles including keywords "survey" and "immigrants" in the selected period (2008 – 2018) included 1.091 articles. After initial study selection according to thematic relevance, we shall analyse the articles in order to detect issues referring to sampling methods and approaching the immigrant population, as well as to identify challenges and limitations of specific quantitative data collection modes. Therefore we will focus on research limitations identified and reported by the authors of relevant studies. The proposed scoping study aims to improve our understanding of various obstacles and challenges researchers encounter in surveying immigrant populations.
Dr Stephanie Steinmetz (University of Amsterdam) - Presenting Author
Ms Janna Besamusca (University of Amsterdam)
The literature shows that hard-to-reach populations are not accurately represented in general population surveys. There are, systematically, higher non-response rates in cross-sectional surveys due to fewer contacts and more refusals as well as higher attrition rates in longitudinal surveys. Various mechanisms for the underrepresentation of those populations have been identified leading to the problem of biased and less reliable parameters.
The aims of this paper are threefold: in a first step, we want to provide an overview of how hard-to-reach populations (such as refuges, LGBT, elderly, but more importantly their intersection with other relevant strata, such as gender and class) can be identified and harmonized across 25 selected EU wide micro-data sets (such as the EU-LFS, EU-SILC, ESS, EVS, ISSP). In a second step, the data quality (related to coverage and sampling as well as item and unit non-response) will be examined by comparing the above mentioned probability surveys with a non-probability web survey (the WageIndicator); and by looking in how far mode effects (face-to-face, telephone and web) between the probability surveys can be identified and related to different levels of data quality. Finally, the paper will address in how far cross-national differences in data quality can be observed when using either different modes of probability-based surveys or a non-probability web survey.
Overall, the paper will identify those existing cross-national data sets that are most suitable for the analysis of hard-to-reach populations.
Dr Snezha Kazakova (Ipsos)
Dr Allison Dunne (Ipsos)
Dr Femke De Keulenaer (Ipsos) - Presenting Author
While public services research is often carefully designed to achieve a representative sample, exclusion of hard-to-reach citizens is a common issue. This case study draws on a large-scale international survey conducted in 2016 for the European Commission to critically examine the use of a mixed mode approach for surveying vulnerable and isolated people.
The study’s objective was to map the health needs and risk factors faced by nine different categories of isolated and vulnerable respondents, including long-term unemployed, the homeless, persons living in isolated areas and victims of domestic violence. Therefore, this case study allows us to analyse the relevance of the mixed mode approach for different groups of hard-to-reach respondents.
The mixed mode approach consisted of two complementary data collection modes. About half of the sample was collected via pen-and-paper surveys that were administered by community organisations (such as social workers and NGOs) recruited specifically for this study. Through their close relationships with the study subjects, they were able to reach vulnerable and isolated respondents that are unlikely to participate in other types of surveys due to a lack of accessibility, mistrust in researchers or a need for assistance due to physical and/or psychological limitations. However, not all respondent categories could be reached via community organisations. To survey hard-to-reach groups, such as vulnerable single parents, the in-work poor, persons living in isolated areas or long-term unemployed, the survey was distributed via a consumer panel.
This presentation will provide an in-depth evaluation of the benefits and drawbacks of this approach in terms of representativity, relevance for different target groups and data validation. This analysis also investigates the differences in survey responses depending on the survey mode controlling for the specific target groups. The discussion will conclude with the key challenging for applying this method for
Mr Jan-Lucas Schanze (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Refugees, elderly, or prisoners belong among others to the institutional populations and could be considered as classical hard-to-survey respondents. As a group, they are a very distinctive, but a comparably small population in most Western countries, what makes them hard-to-sample. Interviewers need to bypass gatekeepers working for the institutions, what makes them hard-to-reach. And depending on the type of institution, language or cultural barriers might arise (e.g., in refugee accommodations) or some respondents might be cognitively impaired (e.g., in retirement and nursing homes), what makes them hard-to-interview. Those reasons lead to an exclusion of institutional residents from the target population in most social surveys and even some health and aging surveys.
The EU-financed SERISS project investigated for more than three years whether it is necessary (due to the peril of bias) and feasible to include institutional populations into surveys of the general population. The presentation draws upon research and main lessons from the SERISS subtasks, mainly coming from a survey inventory of 150 surveys that included institutional residents in Europe, Australia, Canada, and the USA, as well as on an expert survey with more than 40 survey researchers whose survey programs aimed to cover institutional residents.
On the basis of the final report of the SERISS project, the presentation showcases potential tools and means to sample, contact, and interview institutional residents with standardized procedures. It elaborates on potential sampling frames and sampling approaches like oversampling, presents examples on how to address gatekeepers and respondents, and proposes adaptations of standardized interviewing techniques to improve data quality. Moreover, the presentation encourages survey practitioners to reflect whether issues arising in institutions also occur among respondents living in private households. In this case, survey methods adjusted to institutional respondents might also help to improve survey coverage and data quality of hard-to-survey