Sampling Rare and Hard-to-Reach Populations via Social Media - Insights, Challenges and Applications |
|
| Session Organisers | Mrs Zaza Zindel (Bielefeld University) Mr Simon Kühne (Bielefeld University) |
| Time | Friday 2 July, 13:15 - 14:45 |
Researchers face declining response rates and rising survey costs in many countries, making it more challenging to recruit survey participants on an adequate scale - even more challenging when sampling and surveying rare or hard-to-reach populations. In these cases, alternative sampling and recruiting approaches are usually required, including non-probability and online convenience sampling. A rather novel approach to recruiting rare populations for online and mobile survey participation uses advertisements on social media platforms. While some first research findings are promising – e.g., that social media can reduce survey costs, increase overall participation and reach members of the population that would otherwise stay unobserved – aspects of survey error remain rather unclear.
The proposed conference session offers the opportunity to exchange expertise and new developments in social media sampling to improve practices and develop standards.
Keywords: Social Media, Web Survey, Survey Recruitment, Survey Research, Social Media Sampling
Mr André Grow (Max Planck Institute for Demographic Research) - Presenting Author
An increasing number of scholars advocate the use of Facebook’s advertising platform to recruit participants for survey research. This approach has been applied in the context of both surveys of the general population and surveys of hard-to-reach subpopulations. In this presentation, I will discuss this new approach to respondent recruitment in some detail, focusing on advantages, challenges, and practical considerations.
My talk will be based on the COVID-19 Health Behaviour Survey (CHBS). The CHBS is a web-based survey that focuses on people's reactions to the COVID-19 pandemic and targets individuals aged ≥18 years in eight countries (Belgium, France, Germany, Italy, the Netherlands, Spain, the United Kingdom, and the United States). The questionnaire has four sections, which encompass sociodemographic characteristics (e.g., age, sex, and education), health indicators (e.g., underlying medical conditions), behaviours and attitudes related to COVID-19 (e.g., perceived threat level and preventive measures taken), and social contacts (i.e., the number of interactions with other people). Recruitment took place via Facebook, by means of advertisement campaigns that ran on a daily basis between March 13 and April 12, 2020. Within each country, the advertising campaigns were stratified by sex, age group, and region of residence in the respective country, leading to a total of 296 strata across countries. In total, the advertising campaigns resulted in 144,034 completed questionnaires over the entire survey period.
The nature of the COVID-19 pandemic created special circumstances that also affected the sampling process. To illustrate this, I will discuss how the ebbs and flows of the pandemic were related to Facebook user activity and survey participation rates. Second, I will discuss the efficiency of our targeted advertisements in recruiting members of the different strata.
Mr Steffen Pötzschke (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Mr Bernd Weiß (GESIS - Leibniz Institute for the Social Sciences)
Research on international migrants has seen a sharp increase during the last decades; however, most extant studies focus on immigrants. This means that researchers usually limit their analysis to a single or a small number of countries and investigate how newcomers fare in them, what consequences their arrival has for the respective societies, etc. In contrast, research that takes specific groups of emigrants into view and studies them in a great variety of countries is scarce. To a large degree, this is due to the methodological, logistical, and financial challenges associated with cross-national sampling and surveying of populations that are already hard-to-reach when focussing on a single country.
Building on preliminary results of the German Emigrants Overseas Online Survey (GEOOS), we investigate how advertisements on Facebook and Instagram can be employed to reach and subsequently survey emigrants on an almost global scale. Within a field period of only four weeks (August-September 2020) and using a sampling budget of under 2,500€, this study collected data on more than 3,500 Germans in over 140 countries. A main goal of this study consisted in testing the potential of our sampling approach to amend existing methods. Therefore, we focussed on countries beyond the European continent and indeed were able to also realise subsamples of several hundred participants in Latin America, Asia, and Africa; hence in regions where previous surveys on German emigrants struggled to recruit participants (cf. Ette et al., 2020; Pfeiffer & Heimer, 2007).
The contribution provides detailed information on our sampling approach, including measures taken to stratify the sample in terms of age, gender, and geographic distribution. We, furthermore, investigate to which degree the surveyed emigrants could have been reached with established methods, namely by using records of German population registers and through telephone-based sampling approaches. Additionally, we will detail on the sociodemographic composition of our sample and provide some first pointers as to how it compares to those of other surveys. Where appropriate, we compare methodological aspects and substantive findings between world-regions.
Mrs Marieke van der Gaag (Department of Political Science, University of Amsterdam) - Presenting Author
Mrs Evelyn Ersanilli (Department of Political Science, University of Amsterdam)
Sampling migrants is notoriously difficult. Previous studies used a wide-range of sampling strategies. Each of these strategies has its strengths and weaknesses. This paper discusses the sampling strategy for migrants in the MOBILISE project (www.mobiliseproject.com). The MOBILISE project examines why some people respond to discontent by protesting, others by migrating while yet others stay immobile. The migrants survey targets migrants from four countries that have seen outmigration and protest in recent years (Ukraine, Poland, Morocco and Argentina) who live in Germany, the United Kingdom and Spain. Migrants were sampled through Facebook ads linking to an online survey. The survey has two waves with a one year interval. In this paper we will discuss 1) which ad strategies we used and the success of these strategies, 2) participation and attrition bias in the second wave, and 3) offer a primary assessment of the quality of the sample by comparing it to population statistics — where available. We find that sampling through Facebook advertisements is a cost-effective way to obtain a large sample. The method seems particularly effective in reaching recent migrants and reaching migrants from small communities. There is some indication of a bias for gender, education and political interest. Compared to the data collected by Pötzschke & Braun (2017) who surveyed Polish migrants through Facebook ads in 2015-2016, we find that biases have remained similar but that costs of this method have increased – possibly due to lower Facebook use. While willingness to participate in the second wave of data collection was high, actual participation was considerably lower. The paper ends with recommendations on the use of Facebook ads for future migrant surveys.
Mrs Zaza Zindel (Bielefeld University) - Presenting Author
Mr Simon Kühne (Bielefeld University)
A growing portion of the global population is active on social media platforms. This opens up new opportunities for survey research. A rather novel approach is to use online social media and networks like Facebook or Instagram to recruit web survey participants. These platforms are a promising resource to promote surveys and recruit participants, especially if the survey is targeted at a rare population. They provide relatively inexpensive access to a large number of potential respondents and often provide a wealth of additional information to identify and address otherwise hard-to-reach populations. However, there are a number of issues and challenges associated with this type of data collection. First, under-coverage and self-selection into non-probability samples limit inference from the sample to the target population. In addition, there are considerable differences between the individual social media platforms, and the choice of a platform is a decisive factor in determining the audience that can be reached. Finally, it is open whether, and if so, how samples recruited through social media can be adjusted to reflect the distributions of the general population.
This talk will provide insights into the opportunities and hurdles that social media platforms offer for survey research. Various social media samples from the field of social research will be evaluated, with particular attention to survey error and potential data quality issues. Furthermore, by comparing a social media sample with a simultaneously collected, probability-based, face-to-face sample, we assess the potentials, appropriateness and limitations of social media recruitment compared to traditional sampling approaches.
Mr Michael Stern (Michigan State University) - Presenting Author
Mr Erin Fordyce (NORC at the University of Chicago)
Mrs Rachel Carpenter (NORC at the University of Chicago)
Mrs Melissa Heim Viox (NORC at the University of Chicago)
Mrs Sabrina Avripas (NORC at the University of Chicago)
Mr Stuart Michaels (NORC at the University of Chicago)
The use of social media recruitment for surveys has become widespread within the social sciences. Reasons for the upsurge include the relative affordability of posting online advertisements coupled with the general acceptance of nonprobabilistic sample methods among researchers and government offices alike. As a result, within a decade social media recruitment went from virtually nonexistent to a regular tool in the survey researcher’s arsenal. Now, however, we are at a crossroads where we know something about how the demographic characteristics of respondents depend heavily on the source of sample (social media site), method of recruitment (advertisements with or without a snowball or respondent driven sampling approach), and ad design (use of pictures, videos, keywords, hashtags). Questions remain, however, regarding the quality of data collected using this method of recruitment.
This talk will cover several experiments conducted across various projects over the past 8 years to examine aspects of data quality where we used social media recruitment. We will focus on how the choice of social media site, design decisions such as the format of the advertisement (e.g., video or static), targeting, and the use of eligibility language, and device usage impact the quality of the data as measured by demographic variability in comparison to national benchmarks, break-off rates, and the use of non-substantive responses.