All time references are in CEST
Merits and limits of respondent recruitment through social media |
|
| Session Organisers | Professor Simon Kühne (Bielefeld University) Professor Jan Karem Höhne (DZHW, Leibniz University Hannover) Mrs Jessica Donzowa (Max Planck Institute for Demographic Research) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
The demand for high-quality data from surveys – especially from web surveys – is at an all-time peak and continues to grow. At the same time, (web) surveys struggle with low response rates and researchers constantly look for innovative and cost-efficient ways to sample, contact, and interview respondents. One promising approach makes use of social media platforms, such as Facebook, Instagram, and TikTok, for respondent recruitment. Utilizing sophisticated advertisement and target systems offered by these platforms, research has shown that social media recruitment provides an easy, quick, and inexpensive access to an unprecedented and diverse respondent pool (including rare and hard-to-reach populations). Nonetheless, many open questions remain with respect to sample representation, recruitment and advertisement strategies, and data quality and integrity.
In this session, we welcome contributions that are based on empirical studies as well as methodological and theoretical considerations dealing with respondent recruitment through social media platforms. This includes the following research areas (but not limited to):
- Comparing social media platforms for respondent recruitment in terms of costs, quality, and field phase management
- Ad design, invitation wording, and participation metrics
- Strategies for incentivizing respondents and payout
- Targeting designs and sample compositions
- Strategies to cope with user comments and undesired user interactions with ads
- Approaches to deal with missing data in the form of nonresponse and dropouts
- Measurement quality in terms of reliability and validity
- Threat of bots and fake interviews for data integrity
- Comparing social media, nonprobability, and probability-based samples
- Weighting procedures to increase representation and generalizability
- Replications of empirical studies and findings
Keywords: Social media recruitment, Data quality, Data integrity, Representation, Sample comparisons
Mrs Zaza Zindel (German Centre for Integration and Migration Research) - Presenting Author
Social media recruitment has transformed survey research, offering efficient and cost-effective access to diverse and hard-to-reach populations. However, this method also presents significant challenges, such as satisficing, low-effort participation, and identity misrepresentation-issues that remain understudied in social media-recruited surveys. These behaviors pose substantial threats to data qualityand raise important questions: To what extent do such behaviors occur in social media-recruited surveys, and how do they affect analytical results?
This study examines data from a web survey on labor market discrimination against women wearing headscarves in Germany, conducted in 2021 and repeated in 2024. Recruitment was conducted through targeted Facebook ads designed to reach diverse respondents. Problematic responses were identified using indicators such as item non-response, straight-lining, speeding, and identity misrepresentation. Differences between problematic and non-problematic respondents were analyzed using statistical tests (e.g., chi-square, Mann-Whitney U), and multivariate regression models were used to assess the impact of these behaviors on key outcomes, such as perceived anti-Muslim discrimination.
The results reveal systematic differences between problematic and non-problematic respondents. For example, younger and less educated participants were more likely to engage in behaviors such as speeding and straight-lining, leading to distorted results in multivariate analyses. Data cleaning significantly improved the robustness of the findings, highlighting the critical importance of quality control measures in surveys recruited via Facebook.
This study provides a detailed examination of the prevalence, predictors, and consequences of problematic response behavior in social media-recruited surveys. It offers practical recommendations for survey practitioners, emphasizing the need to incorporate quality assurance strategies to enhance data validity and reliability.
Dr Manuela Scioni (Department of Statistical Sciences - University of Padua) - Presenting Author
Dr Margherita Silan (Department of Statistical Sciences - University of Padua)
Social media surveys have become increasingly popular research tools due to their cost-effectiveness, wide reach, and ability to quickly gather diverse opinions. Platforms like Facebook and Instagram offer researchers access to large, demographically varied participant pools with real-time data collection capabilities. However, these surveys face significant challenges, including potential bias from self-selected participants and the risk of under-sampling demographics less active on social media.
This work aims to provide comprehensive guidelines for conducting surveys via social media, with a specific focus on research carried out implementing a quota sampling through advertisements on the Meta platform, offering researchers practical strategies to maximize representativeness of the sample. Using a survey on Internet and Social Media use among the population over 60 years old as a case study, different budget allocation methods to the quotas will be compared both in terms of sample representativeness and in relation to the observation of three benchmark variables in a nationally representative survey. Specifically, three scenarios of budget allocation to the different demographic groups will be compared: a) proportional to the reference population, b) proportional to the population present on the social media platform used, and c) proportional to the response rate obtained in that population segment in a previously conducted pilot survey. Furthermore, various methods for calculating the response rate will be investigated, considering the different metrics proposed by the Meta platform to measure the success of an advertising campaign. The inclusion of the proportions of reach, impressions, unique outbound clicks, accesses to the survey, and completed surveys in the different demographic segments represented by ad-sets in the budget allocation strategy represents an original contribution to strategies to allocate budget Moreover, with the goal of including a heterogeneous sample of respondents, some experiments with different AI-generated images for the adv.
Professor Jan Karem Höhne (DZHW, Leibniz University Hannover) - Presenting Author
Mr Joshua Claassen (DZHW, Leibniz University Hannover)
Mr Ben Wolf (DZHW, Leibniz University Hannover)
Cost- and time-efficient web surveys have progressively replaced other survey modes. These efficiencies can potentially cover the increasing demand for survey data. However, since web surveys suffer from low response rates, researchers and practitioners consider social media platforms as new recruitment source. Although these platforms provide advertisement and targeting systems, the data quality and integrity of web surveys recruited through social media might be threatened by bots (programs that autonomously interact with systems). Bots have the potential to shift survey outcomes and there is literature on how bots infiltrate social media platforms, distribute fake news, and possibly skew public opinion. However, established strategies to stop and detect bots in web surveys are not reliable. This especially applies to LLM-driven bots that are connected to Large Language Models (LLMs). In this study, we therefore focus on LLM-driven bots and investigate whether prompt injections (instructions to change LLM-driven bot behavior) can help to detect bot infiltration in web surveys. To this end, we utilize two bots that are linked to Google’s LLM Gemini Pro and that have different capabilities (e.g., personas and memory). We instructed our two LLM-driven bots to respond to an open question 800 times. This question included either no injection (control), a jailbreaking injection (instructing the LLM to give a specific response), or a prompt leaking injection (instructing the LLM to reveal its prompt). In a next step, we analyze the synthesized data from our LLM-driven bots evaluating the efficiency of the prompt injections under investigation. By focusing on LLM-driven bots, our study stands out from previous studies that mostly focused on conventional, rule-based bots. The investigation of prompt injections potentially extends the methodological toolkit to protect web surveys recruited through social media platforms from LLM-driven bot infiltration.
Mr Dennis Klinke (Freie Universität Berlin) - Presenting Author
Dr Jannes Jacobsen (DeZIM)
Mr Manuel Dierse (Freie Universität Berlin)
Professor Thorsten Faas (Freie Universität Berlin)
Professor Denis Gerstorf (Humboldt-Universität zu Berlin)
Ms Hannah Helal (Freie Universität Berlin)
Professor Swen Hutter (Freie Universität Berlin)
Dr David Schieferdecker (Freie Universität Berlin)
Professor Hanna Schwander (Humboldt-Universität zu Berlin)
Professor Christian von Scheve (Freie Universität Berlin)
Non-probability samples have become increasingly popular despite some criticism. This paper examines social media sampling as a tool for accessing hard-to-reach populations. Our study targeted individuals in Germany using ads on Facebook and X, yielding 4,590 survey respondents. We compared these samples with high-quality probability samples (SOEP, ESS) through measurement equivalence analysis of shared measures between samples. Results show social media sampling can effectively oversample engaged individuals, in our case by ratios between 2:1 and 5:1. Our findings suggest that nonprobability sampling can be a viable method for specific subpopulations when some level of homogeneity for the target population can be expected. As we observe varying levels of measurement equivalence, rigorous methodological strategies for post hoc analyses are recommended. We offer guidelines for designing measurement and sampling strategies that allow for suitable post hoc analysis of measurement equivalence to ensure the fit for purpose of the employed sampling strategy.
Ms Eszter Sandor (Eurofound) - Presenting Author
Mr Michele Consolini (Eurofound)
Eurofound launched a non-probability online survey in March 2020 aiming at measuring the quality of life and work during the early pandemic (Living, working and COVID-19) with a quick turnaround time. The survey was promoted on Meta platforms via a conversion campaign, using COVID-19-related pictures and messages. The survey was repeated four times during the pandemic, and a panel of regular respondents was also established. In spring 2023, a sixth wave was launched, with the pandemic no longer in focus, titled Living and working in the EU. Pixel technology was abandoned due to potential data protection issues, and a traffic campaign was implemented. Instead of pandemic-related imagery, advertisement pictures and messages were tailored to the main contents of the survey. Response to the survey was much lower among social media respondents.
To address this issue, in spring 2024 (wave 7), Eurofound implemented a new advertising strategy, based on advice from previous research on social media advertising. Advertisement sets (assets) were developed for four distinct topics, each with a salient and non-salient image and message combination, as well as a final neutral and general advertisement, so altogether nine assets were launched. Advertisements were placed on the Meta Newsfeed only instead of automatic placement, only static images were used, and ads were placed on an additional platform (LinkedIn).
This presentation compares the sample size and composition achieved in the 7th wave with previous waves, and the number and types of respondents reached by each of the nine advertisements and the two social media platforms.