Factors Associated with Daily Completion Rates in a Smartphone-Based Ecological Momentary Assessment Study

Yong Sook Yang; Gi Wook Ryu; Mona Choi

doi:10.4258/hir.2019.25.4.332

Healthc Inform Res > Volume 25(4); 2019 > Article

Yang, Ryu, and Choi: Factors Associated with Daily Completion Rates in a Smartphone-Based Ecological Momentary Assessment Study

Original Article

Healthcare Informatics Research 2019;25(4):332-337.

Published online: October 31, 2019

DOI: https://doi.org/10.4258/hir.2019.25.4.332

Factors Associated with Daily Completion Rates in a Smartphone-Based Ecological Momentary Assessment Study

Yong Sook Yang

, Gi Wook Ryu

, Mona Choi

Mo-Im Kim Nursing Research Institute, College of Nursing, Yonsei University, Seoul, Korea.

Corresponding Author: Mona Choi. Mo-Im Kim Nursing Research Institute, College of Nursing, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea.
Tel: +82-2-2228-3341, monachoi@yuhs.ac

Received September 16, 2019 Revised October 19, 2019 Accepted October 19, 2019

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Objectives

Ecological momentary assessment (EMA) methods are known to have validity for capturing momentary changes in variables over time. However, data quality relies on the completion rates, which are influenced by both participants' characteristics and study designs. This study applied an EMA method using a mobile application to assess momentary moods and stress levels in patients with Moyamoya disease to examine variables associated with EMA completion rates.

Methods

Adults with Moyamoya disease were recruited from a tertiary hospital in Seoul. Patients with cognitive impairment were excluded. The EMA survey was loaded as a mobile application onto the participants' personal smartphones. Notifications were sent at semi-random intervals four times a day for seven consecutive days. Daily completion rates were calculated as the percentage of completed responses per day; overall completion rates were calculated as the proportion of completed responses per total of the 28 scheduled measures in the study and assessed through a descriptive analysis, t-test, ANOVA, and regression analysis, with mixed modeling to identify the point at which the daily completion rate significantly decreased.

Results

A total of 98 participants responded (mean age, 41.00 ± 10.30 years; 69.4% female; 75.5% married). The overall completion rate was 70.66%, with no gender or age differences found. The daily completion rate decreased significantly after day 5 (p = 0.029).

Conclusions

Obtaining a good completion rate is essential for quality data in EMA methods. Strategic approaches to a study design should be established to encourage participants throughout a study to improve completion rates.

Keywords: Data Accuracy, Ecological Momentary Assessment, Epidemiologic Factors, Guideline Adherence, Mobile Applications

I. Introduction

Experience sampling method or ecological momentary assessment (EMA) methods are known to have validity in capturing momentary changes in variables over time when the assessments are carried out in real-life environments at multiple times [1 2]. This method involves sending alarm reminders to participants to answer questions about their mood or stress several times throughout a study. Recently, EMA has been extensively used for measuring mood and stress in various clinical settings and populations [3].

The quality of EMA data relies on patients' compliance with the study protocol [4]. According to a recent review study that examined predictors of completion rates for patients with chronic pain, completion rates in the EMA method were affected by participants' characteristics such as age and gender [5]. Compliance was also related to EMA design characteristics such as study duration, frequency of response reminders, and length of assessment items. Studies with longer duration, more frequent requests, and lengthier response requirements have been associated with lower completion rates [5 6 7].

However, few studies have evaluated factors associated with EMA completion rates when measuring mood and stress in adult patients with Moyamoya disease, a chronic progressive cerebrovascular disease with unknown cause. In the current study, we measured the momentary mood and stress of patients with Moyamoya disease using the EMA method as a mobile application developed in our previous study of the general population [8]. This study aims to examine factors affecting completion rates of EMA using a mobile application for adult patients with Moyamoya disease.

II. Methods

1. Sample

We recruited adult patients over 18 years of age with Moyamoya disease from a tertiary university hospital in Seoul, Korea. Participants whose mobile phones used the iOS operating system were excluded because the application is currently available only for the Android operating system [8]. Cognitively impaired participants, as evaluated by the Korean version of the Mini-Mental State Examination [9], were also excluded. This study was approved by the Institutional Review Board of the Yonsei University Health System (No. 4-2018-0385).

2. Measurement

A structured questionnaire was used to collect demographic characteristics and disease-specific information from the patients. The EMA survey included the 13-item PsyMate [10 11 12 13 14] for measuring mood and the 8-item Trier Inventory for Chronic Stress (TICS), adapted from the 9-item TICS [15 16 17], with exclusion of “chronic worrying” since it was intended to assess momentary stress. PsyMate was rated on a 7-point scale (1 to 7) and TICS on a 5-point scale (1 to 5). Detailed information on developing the process of the tools is provided in a previous work [8].

3. EMA Procedure

After signing the informed consent form, all participants downloaded and enrolled in the mobile application using their Android-based mobile phones with the help of the research team. Participants were provided practice time to learn how to answer the EMA application, with the research team helping as needed. Participants were informed that for seven consecutive days they would be notified with an alarm beep when it was time to answer. Alarms were sent during four time windows each day: 8–9 am, 12–1 pm, 5–6 pm, and 9–10 pm at semi-random intervals during the 60-minute blocks. Reminder beeps were sent 45 minutes after the first alarm for participants who had not answered.

Responses from the participants were closely monitored through the application by the research team. When participants missed all four responses of the first EMA day, they were telephoned on the second day to determine the reasons for their unresponsiveness, such as technical problems or intention to drop out. Participants were advised to contact the research team freely with any questions they might have during the survey.

4. Statistical Analysis

We applied descriptive statistics to analyze the personal and disease-specific characteristics of participants using SPSS v.25.0 (IBM SPSS, Armonk, NY, USA) software. Overall completion rates were calculated as the proportion of completed EMA responses out of the total of 28 scheduled measures during the study. We performed descriptive analysis, t-test, ANOVA, and regression analysis to examine the associated factors influencing the overall completion rates. Additionally, we applied the Mann–Whitney U test and the Kruskal–Wallis test to confirm the results from the parametric analysis since the dependent variable was not normally distributed.

Daily completion rates were calculated as the percentage of completed responses for each day and compared between days to examine changes over the 7-day study duration. Since the data were clustered and correlated due to the repeated measurements, we performed mixed modeling to identify the day on which the daily completion rate significantly decreased compared to the first day.

III. Results

1. Sample

There were 98 adult patients with Moyamoya disease in the EMA survey—mean age, 41.00 ± 10.30 years; 68 (69.4%) females; and 74 (75.5%) married. Among these patients, 54 (55.1%) were college graduates or beyond, and 57 (58.2%) were employed. The mean of the perceived severity score was 3.53 ± 0.99 out of 5, and the mean of years since diagnosis was 3.65 ± 4.00 years. Demographic and disease-specific characteristics of the participants are presented in Table 1.

2. Overall Completion Rates

The overall completion rate, calculated as the proportion of individual responses out of the total of 28 scheduled measures, was 70.66%, with the individual completions rates ranging from 3.57% to 100%. The distribution of overall completion rates is shown in Figure 1.

There was no mean difference in the overall completion rates by age, sex, job, or participants' disease-specific characteristics of perceived severity or years since diagnosis. No variables associated with the overall completion rates were identified by the regression analysis. The overall completion rates by demographic and disease-specific characteristics are listed in Table 2.

3. Daily Completion Rates

Daily completion rates for the seven days, calculated as the percentage of completed responses for each day, ranged from 57.40% to 81.12%. We found no difference in daily completion rates according to the day of the week (i.e., weekdays vs. weekend) or time of the day. The mean difference of the daily completion rates between days were significant in ANOVA. Daily completion rates by sequential days are shown in Figure 2 and Table 3.

We performed a mixed modeling analysis to identify the day point when the daily completion rate significantly decreased. On day 5, the daily completion rate was significantly lower (coefficient = −7.40, p = 0.029) than on the first day. It was also lower on day 6 (coefficient = −18.37, p < 0.001) and 7 (coefficient = −16.07, p < 0.001) than on the first day. The estimated parameters are presented in Table 4.

IV. Discussion

This study evaluated how participants' personal variables and study designs affected the completion rates of an EMA-method self-assessment of momentary mood and stress levels among adult patients with Moyamoya disease. There is no agreed upon gold standard for an acceptable completion rate for analyzing data collected by the EMA method. Often, the one-third rule is used as a rule of thumb for analyzable data, meaning that participants with fewer than one-third valid responses out of the total are excluded from the analyses [18]. In the current study, 84 participants (85.7%) responded more than one-third of the time (eight times). However, we included all available participants, regardless of the individual completion rates, since this study aimed to evaluate the associated factors affecting completion rates.

The overall completion rate of the study was 70.66% (range, 3.57%–100%), which was lower than that of studies of end-stage kidney disease patients (86%) [19] and of HIV patients (81.3%) [20] for the same study duration (7 days). However, it was higher than that of 6-day studies for patients with chronic fatigue syndrome [21 22]. The overall completion rates did not differ by sex or age, and this result was similar to that of one previous study [23] but different from the results of several studies that showed lower completion rates among men [6 7] and younger participants [7]. These different results from other studies imply that overall completion rates may vary not only by participants' demographic characteristics but also by personal attitude or perception toward mobile applications. Disease-specific characteristics such as years since diagnosis and perceived severity were not associated with the overall completion rate, and this result was similar to that of a previous review study for patients with chronic pain [7].

Daily completion rates were found to depend on EMA study design characteristics such as study duration, frequency of response, and intensity of measures [7]. In the current study, we measured four times a day for 7 days. The daily completion rate decreased significantly from day 5 compared with day 1. This result is in line with a study that reported that long study durations were associated with low daily completion rates [7]. Additionally, the daily completion rate of the second day was slightly increased compared to the first day; this might have been associated with the phone calls from the researcher team on day 2, although the purpose of the phone calls was to check for any technical problems, difficulties, or intention to drop out.

Alarm reminders were delivered four times a day during 60-minute blocks (8–9 am, 12–1 pm, 5–6 pm, and 9–10 pm). The completion rate of time four (the block between 9 and 10 pm) was lower than that for other times, but the difference was not significant (p = 0.072). Nonetheless, this needs to be addressed since not everyone might have been awake during the study hours since participants were instructed to follow their own daily circadian rhythms. Moreover, in the current study, participants used their own smartphones, with which they were very familiar. The reminder beeps thus could easily have been missed since the participants might have been so accustomed to their phones' sounds that they had learned to ignore them at busy times. This would have had consequences for the completion rates, as identified in a previous EMA study using an electronic diary [24].

This study had limitations. The current study included patients who responded to the EMA study, and those who did not respond were excluded although they initially agreed to participate. Another limitation is that reasons for dropping out during the EMA survey period were not evaluated with the exception of a few patients who gave feedback noting they were too busy to actively participate. In-depth individual or focus group interviews are a potentially useful method to assess factors associated with completion rates for future studies.

Considering the known variables related to study design such as EMA duration, frequency, and alarm times, strategic approaches should be established to improve completion rates and the quality of data in EMA studies. Additionally, participant-centered strategies for building positive relationships and communication among participants and researchers must be prepared to motivate and encourage participants to remain involved throughout the study duration of a longitudinal EMA study.

Acknowledgments

We thank Simone Verhagen, Department of Psychiatry and Neuropsychology at Maastricht University, for sharing her experience using EMA methods. This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (No. 2017R1D1A1B03030706).

Notes

Conflict of Interest: Mona Choi is an editor of Healthcare Informatics Research; however, she did not involve in the peer reviewer selection, evaluation, and decision process of this article. Otherwise, no potential conflict of interest relevant to this article was reported.

References

1. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008;4:1-32. PMID: 18509902.

2. Augustine AA, Larsen RJ. Emotion research. In: Mehl MR, Conner TS, editors. Handbook of research methods for studying daily life. New York (NY): Guilford Press; 2012. p. 497-510.

3. Yang YS, Ryu GW, Choi M. Methodological strategies for ecological momentary assessment to evaluate mood and stress in adult patients using mobile phones: systematic review. JMIR Mhealth Uhealth 2019;7(4):e11215. PMID: 30932866.

4. Stone AA, Shiffman S. Capturing momentary, self-report data: a proposal for reporting guidelines. Ann Behav Med 2002;24(3):236-243. PMID: 12173681.

5. Courvoisier DS, Eid M, Lischetzke T. Compliance to a cell phone-based ecological momentary assessment study: the effect of time and personality characteristics. Psychol Assess 2012;24(3):713-720. PMID: 22250597.

6. Messiah A, Grondin O, Encrenaz G. Factors associated with missing data in an experience sampling investigation of substance use determinants. Drug Alcohol Depend 2011;114(2-3):153-158. PMID: 21075563.

7. Ono M, Schneider S, Junghaenel DU, Stone AA. What affects the completion of ecological momentary assessments in chronic pain research? An individual patient data meta-analysis. J Med Internet Res 2019;21(2):e11398. PMID: 30720437.

8. Yang YS, Ryu GW, Han I, Oh S, Choi M. Ecological momentary assessment using smartphone-based mobile application for affect and stress assessment. Healthc Inform Res 2018;24(4):381-386. PMID: 30443427.

9. Kang Y, Na DL, Hahn S. A validity study on the Korean Mini-Mental State Examination (K-MMSE) in dementia patients. J Korean Neurol Assoc 1997;15(2):300-307.

10. Verhagen SJ, Berben JA, Leue C, Marsman A, Delespaul PA, van Os J, et al. Demonstrating the reliability of transdiagnostic mHealth Routine Outcome Monitoring in mental health services using experience sampling technology. PLoS One 2017;12(10):e0186294. PMID: 29023568.

11. Kramer I, Simons CJ, Hartmann JA, Menne-Lothmann C, Viechtbauer W, Peeters F, et al. A therapeutic application of the experience sampling method in the treatment of depression: a randomized controlled trial. World Psychiatry 2014;13(1):68-77. PMID: 24497255.

12. Jacobs N, Menne-Lothmann C, Derom C, Thiery E, van Os J, Wichers M. Deconstructing the familiality of variability in momentary negative and positive affect. Acta Psychiatr Scand 2013;127(4):318-327. PMID: 22906203.

13. Snippe E, Simons CJ, Hartmann JA, Menne-Lothmann C, Kramer I, Booij SH, et al. Change in daily life behaviors and depression: within-person and between-person associations. Health Psychol 2016;35(5):433-441. PMID: 26690641.

14. van Os J, Verhagen S, Marsman A, Peeters F, Bak M, Marcelis M, et al. The experience sampling method as an mHealth tool to support self-monitoring, self-insight, and personalized health care in clinical practice. Depress Anxiety 2017;34(6):481-493. PMID: 28544391.

15. Powell DJ, Schlotz W. Daily life stress and the cortisol awakening response: testing the anticipation hypothesis. PLoS One 2012;7(12):e52067. PMID: 23284871.

16. Petrowski K, Paul S, Albani C, Brahler E. Factor structure and psychometric properties of the trier inventory for chronic stress (TICS) in a representative German sample. BMC Med Res Methodol 2012;12:42. PMID: 22463771.

17. Petrowski K, Bastianon CD, Buhrer S, Brahler E. Air quality and chronic stress: a representative study of air pollution (PM2.5, PM10) in Germany. J Occup Environ Med 2019;61(2):144-147. PMID: 30475313.

18. Delespaul PA. Assessing schizophrenia in daily life: the experience sampling method. Maastricht, the Netherlands: Maastricht University; 1995.

19. Abdel-Kader K, Jhamb M, Mandich LA, Yabes J, Keene RM, Beach S, et al. Ecological momentary assessment of fatigue, sleepiness, and exhaustion in ESKD. BMC Nephrol 2014;15:29. PMID: 24502751.

20. Moore RC, Kaufmann CN, Rooney AS, Moore DJ, Eyler LT, Granholm E, et al. Feasibility and acceptability of ecological momentary assessment of daily functioning among older adults with HIV. Am J Geriatr Psychiatry 2017;25(8):829-840. PMID: 28065496.

21. Band R, Barrowclough C, Emsley R, Machin M, Wearden AJ. Significant other behavioural responses and patient chronic fatigue syndrome symptom fluctuations in the context of daily life: an experience sampling study. Br J Health Psychol 2016;21(3):499-514. PMID: 26700742.

22. Band R, Barrowclough C, Caldwell K, Emsley R, Wearden A. Activity patterns in response to symptoms in patients being treated for chronic fatigue syndrome: an experience sampling methodology study. Health Psychol 2017;36(3):264-269. PMID: 27819461.

23. Okifuji A, Bradshaw DH, Donaldson GW, Turk DC. Sequential analyses of daily symptoms in women with fibromyalgia syndrome. J Pain 2011;12(1):84-93. PMID: 20591745.

24. Aaron LA, Mancl L, Turner JA, Sawchuk CN, Klein KM. Reasons for missing interviews in the daily electronic assessment of pain, mood, and stress. Pain 2004;109(3):389-398. PMID: 15157700.