If I Text, Will You Respond? Mobile Text Messaging During Recruitment and Data Collection |
|
| Session Organisers |
Dr Ipek Bilgen (NORC) Dr Antje Kirchner (RTI International) |
| Time | Thursday 18th July, 14:00 - 15:30 |
| Room | D17 |
As the use of cell phones and smartphones has grown in popularity globally, the use of text messaging (also known as short message service or SMS) in public opinion research has been increasing exponentially within the last couple decades. Due to decreasing response rates, shifts in technology, and technological enhancements around the way people use text messaging, researchers have started exploring new ways to incorporate texting during the survey life cycle. However, these new trends present challenges and researchers are only just beginning to understand the impact of text messaging on total survey error. Accordingly, in this session we welcome contributions on a range of topics related to the use and impact of texting during survey recruitment and data collection from a total survey error perspective. Specifically, we welcome papers including the following topics:
• Issues related to coverage and sample representativeness, including issues related to sampling frames and sample sources for text-based surveys or mixed mode surveys.
• The impact of text recruitment on response (e.g., response rates, nonresponse error and bias) and design decisions related to response including but not limited to
o Tailoring text communications and the content of messages
• Data collection and questionnaire design, including but not limited to
o Measurement
♣ Break-offs and item nonresponse
♣ Measurement limitations
o The use of short message service (SMS) versus multimedia messages (MMS)
o Comparison with other self-administered modes (such as web, mail, IVR)
• Trade-offs and interactions between survey error sources
• Analyses from a survey operations perspective (e.g. day/time of the day, two-way versus one way interactions, etc.)
• Privacy and safety regulations (such as TCPA and GDPR)
• Cross-national and cross-cultural differences
o Implications for the future (the increase in the use of smartphones)
o Implications for developing countries
We are interested in examples from a range of different types of studies including but not limited to mixed mode surveys, online panels, and longitudinal studies that include SMS components. We welcome papers from researchers with a variety of backgrounds across different nations and sectors including academia, government agencies, and industry. The format of the session will be designed to foster discussion between the presenters and audience around issues related to potential uses of SMS methodology within the survey life cycle from a total survey error perspective.
Keywords: SMS, Recruitment, Data Collection, Total Survey Error, Rules and Regulations
Mr Ole-Martin Vangen (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway) - Presenting Author
Mr Knut-Petter Leinan (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway)
Dr Tore Wentzel-Larsen (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway)
Dr Annette Jeneson (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway)
Background
Use of smartphones changes rapidly, and the overall aim of this study is to gain current data on effects of invitation mode to guide recruitment and data collection practices. Previous studies suggest that invitations via text message yields slightly higher response rates compared to invitation via email, especially for respondents who complete the survey on a mobile.
Objective
To explore response rates using mobile Text messages, compared to emailing, to send out survey invitations and reminders.
Methods
This study comparing the effects of two invitation modes for data collection is a “trial within a trial”.
Web surveys are sent to parents of young children who are in day care centres receiving a program aimed to increase the quality of care. To test the effect of this program intervention, parents are invited to answer a 30-minute web survey pre- and post- intervention. The pre-intervention survey is currently active, and we expect approximately 2300 completed surveys. Invitations to the post-intervention survey will be sent May 2019. In the current pre-intervention survey, we ask respondents if they are willing to register a mobile number so that we might send the post-intervention survey invitation via text message. Consenting respondents will be randomized to receive the post-intervention invitation, with survey link, via text message or e-mail. We will also test the effects of these different invitation modes on response to reminders. Stratified randomization will be used to ensure balance of groups with respect to device used to complete the pre-intervention survey (Mobile versus PC; approximately 60% versus 40% in surveys completed so far).
Results
The recruitment is currently ongoing and we expect to have completed our data collection by Spring 2019. So far, 1087 respondents (49,3% of the total number respondents so far) have registered a mobile number and are eligible for this trial.
Ms Hannah Bucher (GESIS) - Presenting Author
Mr Matthias Sand (GESIS)
The strongly increasing number of people who own a mobile device with internet access implements new developments into web-survey-research. This increasing rate also has an impact on the devices used to complete a web-survey. Further, some studies investigate differences in demographics of the respondents associated with the used device; therefore it seems interesting to examine the possibilities of recruiting survey units via mobile phones.
As previous studies either used mobile phones as a dedicated sampling frame for CATI-surveys or used text messages to invite participants of an access panel to take part in specific web-surveys, we combined these approaches and investigate a new sampling approach for web-surveys: The recruitment of people for a Web-Survey via a mobile RDD-mobile phone sampling with an invitation via text message that contains a tiny URL to the questionnaire. This would generally also allow for frequentist estimation. Hence, this paper aims to examine the possibilities of such recruitment methods. This new approach will, therefore, be discussed, regarding times and days of contact. Of particular interest is furthermore the sampling population that can be recruited by this approach as well as its outcome rates and costs.
Preliminary results indicate that this sampling method is not very suitable for questioning the broad population, due to problems occurring at different levels:
On the one hand, considerable technical problems arose in the process of verifying automatically generated mobile phone numbers (HRL-lookup). On the other hand, less than one percent of the sent text messages lead to an actual interview.
Our research is the first to investigate a new probabilistic recruitment strategy for mobile web surveys. Therefore, it contributes substantively to the exploration of new survey recruitment strategies to replace or complement existing recruitment strategies that increased in costs such as face to face surveys or decrease in quality such as
Professor Ioannis Andreadis (Aristotle University of Thessaloniki) - Presenting Author
This paper examines the findings from a smartphone optimized web survey, when all invitations and reminders are sent as text messages via the short message service (SMS) to the mobile phones of the target group. The web survey under study was conducted for the National Theatre of Northern Greece (NTNG), the largest theatrical organization of Greece comprising of five winter venues and two open-air theatres. The list of mobile phone numbers was provided by the organization and the aim of the survey was to collect data from the people who have attended events organized by NTNG.
The paper summarizes the results of an experiment conducted among respondents: several groups of respondents have been created randomly. In each group, the time of the day the invitations and reminders were sent, the length of the period between reminders and the text of the invitations and reminders have been manipulated. The findings suggest that when the mobile phone numbers of the target group are available, although the limited length of the SMS texts makes it more difficult to convince respondents to participate in the survey, it is feasible to conduct a successful large-scale web survey using SMS as the only contact mode. The paper concludes with suggestions based on the experiment findings about the best practices that produce the maximum results for a smartphone optimized web survey when all invitations and reminders are sent via SMS.
Dr Ipek Bilgen (NORC)
Dr Antje Kirchner (RTI International) - Presenting Author
As cell phones and smartphones have grown in popularity globally, the use of text messaging (aka short message service or SMS) in survey research has been increasing exponentially within the last couple decades. Due to decreasing response rates, shifts in technology, and technological enhancements around the way people use text messaging, researchers have started exploring new ways to incorporate texting during the survey life cycle. However, these new trends present challenges and researchers are only just beginning to understand the impact of text messaging on total survey error. As such, this paper investigates in which context(s) text messaging presents a viable alternative to other modes of recruitment and data collection from a total survey error perspective.
Conducting a meta-analysis of published and unpublished studies, we first analyze the impact of text message recruitment and/or data collection on survey response rates and nonresponse error. We also assess the challenges of text message recruitment and data collection associated with coverage and sample representativeness, including issues related to sampling frames and sample sources. We then investigate the effect of text message questionnaire design decisions on measurement error. As part of our analyses we are not only investigating each error source individually but also account for the trade-offs of and interactions between survey error sources also in comparison to other self-administered modes. Additionally, we examine text message use within the context of mixed mode methodology.
To fully understand the potential of text message surveys as a tool for future data collection efforts, the above analysis also includes an assessment of privacy and regulatory issues specific to text messaging (e.g., TCPA and GDPR), and their implications on data quality and efficiency also from a cross-national and cross-cultural perspective. The paper concludes with a set of recommendations regarding the use of text messaging in surveys.