All time references are in CEST
Challenges and opportunities when using Ecological Momentary Assessment (EMA) |
|
| Session Organisers | Professor Florian Keusch (University of Mannheim) Dr Heidi Guyer (RTI International) Professor Bella Struminskaya (Utrecht University) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
The widespread use of smartphones in the general population has led to a renaissance of Ecological Momentary Assessment (EMA), sometimes also called Experience Sampling Method (ESM). The goal of EMAs is to collect intensive longitudinal data with high ecological validity by sending alerts based on predetermined criteria to participants on their smartphone. Participants are then asked to self-report their current subjective (e.g., stress, mood, emotional well-being) and/or objective state (e.g., activities, co-location with others). EMA data can be implemented to obtain additional information beyond survey data, as an evaluation or intervention tool, or on its own. EMA data allow researchers to study between-person differences in daily life across people in various social and physical contexts as well as within-person changes over the course of a day, a week, or another specified period. However, multiple and frequent measurements can be rather burdensome for participants, leading to item nonresponse and attrition. For this session, we invite methodological research that addresses issues of nonparticipation and measurement in EMA studies. We are particularly interested in studies that investigate the effect of different aspects of the research design, for example, the communication with participants, the User Interface of EMA apps, the use of incentives and gamification approaches, providing feedback to participants, etc., on data quality. We also invite presentations of study designs where the requests for self-reports are triggered based on behaviors or states that are inferred from sensors built into smartphones or wearable devices. Such approaches can include the use of geofencing, trip and/or activity detection, smartphone usage behavior-based triggers (e.g., when using a social media app), etc. With the rapid expansion of EMAs in survey research, this session will allow researchers to share their findings and lessons learned to date.
Keywords: EMA, ESM, apps, wearables, data quality
Mr Martin Kreidl (Masaryk university) - Presenting Author
Mr Daniel Dvorak (Masaryk university)
This paper explores how the number of questionnaire notifications (displayed on a smartphone) affects compliance (i.e.: respondent opened/completed the questionnaire) and attrition in an EMA study that distributed 4 short questionnaires per day over the course of 15 days. While most survey practitioners agree that the effect of each notification declines with the total number of notifications issued, some studies indicated that too frequent notifications might even have a negative effect on compliance. The effects of notifications in EMA studies are not well understood and there are no clear methodological guidelines in the literature.
Furthermore, we also explore how the number of displayed notifications affects attrition (drop-out from the EMA study) arguing that the effects of notifications might cumulate over questionnaires within a day or even over days in the study.
We find that the effect of notifications on compliance with a particular questionnaire decline, but never become negative. Yet, across multiple questionnaires, the cumulative effect of the number of notifications does become negative, effectively increasing – albeit slightly – attrition.
Thus, EMA practitioners do face a trade-off: frequent questionnaire notifications increase the odds that a respondent will open/complete a particular questionnaire. Yet, they also increase the likelihood that the respondent will drop-out from the study. Balancing these two contradictory tendencies is a major task when designing any EMA study.
Dr Francisco Perez-Arce (University of Southern California) - Presenting Author
Dr Evan Sandlin (University of Southern California)
Professor Arthur Stone (University of Southern California)
Professor Arie Kapteyn (University of Southern California)
This study examines how major life events influence subjective well-being, focusing on differences between life satisfaction and momentary affect. Using Ecological Momentary Assessment (EMA) data from a nationally representative panel of adults aged 50 and older, we analyze responses to significant life events, including health issues, financial shocks, and bereavements. Participants reported their wellbeing during baseline "random bursts" and subsequent "triggered bursts" following life events. Results reveal that while life satisfaction exhibits significant changes in response to these events, momentary affect remains largely unaffected, with smaller or statistically insignificant effects. Notably, the impacts of participant age and the day of the week on momentary affect exceeded those of life events. These findings support the "focusing hypothesis," suggesting that while life satisfaction may adapt to significant circumstances, day-to-day emotional experiences are more resilient.
Professor Chan Zhang (Zhejiang University) - Presenting Author
Mr Shuaiying Cao (Zhejiang University)
Ms Minglei Wang (Zhejiang University)
Dr Marc Tibber (University College London )
Ecological Momentary Assessment (EMA) is an intensive longitudinal data collection method to capture in-the-moment experiences through frequent assessments. With the advent of mobile technologies, EMA's applications have expanded across a number of disciplines. Despite its growing popularity, methodological issues, particularly regarding response quality, have yet to be explored. This study systemically evaluates how response quality changes over time in a 21-day EMA study with five daily assessments. Data were analyzed from 100 university students who completed surveys via a mobile application. Response quality was measured using a variety of satisficing indicators, including speeding, non-differentiation in grids, reporting "0" to filter questions, and anchoring (i.e., providing the same answer to the question as in the previous assessment). The findings revealed a significant increase in speeding over the three weeks, indicating a decrease in response quality. There was also an increase in reporting "0" to filter questions and anchoring responses in weeks two and three. The non-differentiation in grids mainly stayed the same across the weeks, which might be because the grids in this study only contained a few simple items to rate. The findings also showed variations between participants regarding their response behaviors across the weeks. However, such variation could not be explained by participants’ demographic characteristics or motives for participation. Compared to the changes in response behaviors across weeks, the response quality differences across the five daily assessments were less systematic and inconsistent. These findings highlight the need for effective strategies to motivate participants to provide thoughtful answers as EMA data collection proceeds, and careful consideration of EMA design parameters to ensure high-quality data collection.
Mr Carlos Ochoa (Universitat Pompeu Fabra (RECSM)) - Presenting Author
Memory issues in surveys are well-known yet challenging to address. Over the past several decades, two main strategies have been explored to mitigate the effects of our limited ability to recall past events. One strategy involves replacing retrospective surveys with passively collected data, such as digital trace or geolocation data. However, this approach cannot capture all types of objective or subjective data.
The second strategy reduces the time gap between events and surveys, as seen in fields like audience research and psychology. Ecological Momentary Assessment (EMA), developed in psychology, collects data on individuals’ behaviors and experiences through repeated sampling in real time. Typically, sampling follows a random or predefined schedule, but passive data enable surveys to be triggered by specific detected events. This method, also known as "in-the-moment surveys," expands EMA’s applicability across diverse fields.
This presentation discusses insights from an in-the-moment survey triggered by geolocation data developed the summer of 2024. Members of the Netquest opt-in online panel with continuous geolocation sharing were surveyed upon entering any of 2,480 mainland beaches in Spain. Topics included transportation modes, general beach conditions, bathing and sunbathing behaviors, and attitudes toward sunbathing. Responses from 454 completed in-the-moment surveys were compared with a similar-sized sample from an equivalent retrospective survey.
Direct comparison across participation, user experience, data quality, and substantive results showed that geolocation-triggered surveys conducted on online panels are feasible and well-received. However, contrary to expectations, explicit non-recall ("don’t know" answers) was not significantly reduced. Moreover, differences in substantive answers and comparisons with external data (e.g., beach temperatures recorded by meteorological stations) suggest that retrospective survey participants may reconstruct responses without realizing their memory limitations.
This study highlights the potential and challenges of integrating geolocation data into