PDF
ePub
Citation
Print
I. Introduction
Experience sampling method or ecological momentary assessment (EMA) methods are known to have validity in capturing momentary changes in variables over time when the assessments are carried out in real-life environments at multiple times [12]. This method involves sending alarm reminders to participants to answer questions about their mood or stress several times throughout a study. Recently, EMA has been extensively used for measuring mood and stress in various clinical settings and populations [3].
The quality of EMA data relies on patients' compliance with the study protocol [4]. According to a recent review study that examined predictors of completion rates for patients with chronic pain, completion rates in the EMA method were affected by participants' characteristics such as age and gender [5]. Compliance was also related to EMA design characteristics such as study duration, frequency of response reminders, and length of assessment items. Studies with longer duration, more frequent requests, and lengthier response requirements have been associated with lower completion rates [567].
However, few studies have evaluated factors associated with EMA completion rates when measuring mood and stress in adult patients with Moyamoya disease, a chronic progressive cerebrovascular disease with unknown cause. In the current study, we measured the momentary mood and stress of patients with Moyamoya disease using the EMA method as a mobile application developed in our previous study of the general population [8]. This study aims to examine factors affecting completion rates of EMA using a mobile application for adult patients with Moyamoya disease.
II. Methods
1. Sample
We recruited adult patients over 18 years of age with Moyamoya disease from a tertiary university hospital in Seoul, Korea. Participants whose mobile phones used the iOS operating system were excluded because the application is currently available only for the Android operating system [8]. Cognitively impaired participants, as evaluated by the Korean version of the Mini-Mental State Examination [9], were also excluded. This study was approved by the Institutional Review Board of the Yonsei University Health System (No. 4-2018-0385).
2. Measurement
A structured questionnaire was used to collect demographic characteristics and disease-specific information from the patients. The EMA survey included the 13-item PsyMate [1011121314] for measuring mood and the 8-item Trier Inventory for Chronic Stress (TICS), adapted from the 9-item TICS [151617], with exclusion of “chronic worrying” since it was intended to assess momentary stress. PsyMate was rated on a 7-point scale (1 to 7) and TICS on a 5-point scale (1 to 5). Detailed information on developing the process of the tools is provided in a previous work [8].
3. EMA Procedure
After signing the informed consent form, all participants downloaded and enrolled in the mobile application using their Android-based mobile phones with the help of the research team. Participants were provided practice time to learn how to answer the EMA application, with the research team helping as needed. Participants were informed that for seven consecutive days they would be notified with an alarm beep when it was time to answer. Alarms were sent during four time windows each day: 8–9 am, 12–1 pm, 5–6 pm, and 9–10 pm at semi-random intervals during the 60-minute blocks. Reminder beeps were sent 45 minutes after the first alarm for participants who had not answered.
Responses from the participants were closely monitored through the application by the research team. When participants missed all four responses of the first EMA day, they were telephoned on the second day to determine the reasons for their unresponsiveness, such as technical problems or intention to drop out. Participants were advised to contact the research team freely with any questions they might have during the survey.
4. Statistical Analysis
We applied descriptive statistics to analyze the personal and disease-specific characteristics of participants using SPSS v.25.0 (IBM SPSS, Armonk, NY, USA) software. Overall completion rates were calculated as the proportion of completed EMA responses out of the total of 28 scheduled measures during the study. We performed descriptive analysis, t-test, ANOVA, and regression analysis to examine the associated factors influencing the overall completion rates. Additionally, we applied the Mann–Whitney U test and the Kruskal–Wallis test to confirm the results from the parametric analysis since the dependent variable was not normally distributed.
Daily completion rates were calculated as the percentage of completed responses for each day and compared between days to examine changes over the 7-day study duration. Since the data were clustered and correlated due to the repeated measurements, we performed mixed modeling to identify the day on which the daily completion rate significantly decreased compared to the first day.
III. Results
1. Sample
There were 98 adult patients with Moyamoya disease in the EMA survey—mean age, 41.00 ± 10.30 years; 68 (69.4%) females; and 74 (75.5%) married. Among these patients, 54 (55.1%) were college graduates or beyond, and 57 (58.2%) were employed. The mean of the perceived severity score was 3.53 ± 0.99 out of 5, and the mean of years since diagnosis was 3.65 ± 4.00 years. Demographic and disease-specific characteristics of the participants are presented in Table 1.
2. Overall Completion Rates
The overall completion rate, calculated as the proportion of individual responses out of the total of 28 scheduled measures, was 70.66%, with the individual completions rates ranging from 3.57% to 100%. The distribution of overall completion rates is shown in Figure 1.
There was no mean difference in the overall completion rates by age, sex, job, or participants' disease-specific characteristics of perceived severity or years since diagnosis. No variables associated with the overall completion rates were identified by the regression analysis. The overall completion rates by demographic and disease-specific characteristics are listed in Table 2.
3. Daily Completion Rates
Daily completion rates for the seven days, calculated as the percentage of completed responses for each day, ranged from 57.40% to 81.12%. We found no difference in daily completion rates according to the day of the week (i.e., weekdays vs. weekend) or time of the day. The mean difference of the daily completion rates between days were significant in ANOVA. Daily completion rates by sequential days are shown in Figure 2 and Table 3.
We performed a mixed modeling analysis to identify the day point when the daily completion rate significantly decreased. On day 5, the daily completion rate was significantly lower (coefficient = −7.40, p = 0.029) than on the first day. It was also lower on day 6 (coefficient = −18.37, p < 0.001) and 7 (coefficient = −16.07, p < 0.001) than on the first day. The estimated parameters are presented in Table 4.
IV. Discussion
This study evaluated how participants' personal variables and study designs affected the completion rates of an EMA-method self-assessment of momentary mood and stress levels among adult patients with Moyamoya disease. There is no agreed upon gold standard for an acceptable completion rate for analyzing data collected by the EMA method. Often, the one-third rule is used as a rule of thumb for analyzable data, meaning that participants with fewer than one-third valid responses out of the total are excluded from the analyses [18]. In the current study, 84 participants (85.7%) responded more than one-third of the time (eight times). However, we included all available participants, regardless of the individual completion rates, since this study aimed to evaluate the associated factors affecting completion rates.
The overall completion rate of the study was 70.66% (range, 3.57%–100%), which was lower than that of studies of end-stage kidney disease patients (86%) [19] and of HIV patients (81.3%) [20] for the same study duration (7 days). However, it was higher than that of 6-day studies for patients with chronic fatigue syndrome [2122]. The overall completion rates did not differ by sex or age, and this result was similar to that of one previous study [23] but different from the results of several studies that showed lower completion rates among men [67] and younger participants [7]. These different results from other studies imply that overall completion rates may vary not only by participants' demographic characteristics but also by personal attitude or perception toward mobile applications. Disease-specific characteristics such as years since diagnosis and perceived severity were not associated with the overall completion rate, and this result was similar to that of a previous review study for patients with chronic pain [7].
Daily completion rates were found to depend on EMA study design characteristics such as study duration, frequency of response, and intensity of measures [7]. In the current study, we measured four times a day for 7 days. The daily completion rate decreased significantly from day 5 compared with day 1. This result is in line with a study that reported that long study durations were associated with low daily completion rates [7]. Additionally, the daily completion rate of the second day was slightly increased compared to the first day; this might have been associated with the phone calls from the researcher team on day 2, although the purpose of the phone calls was to check for any technical problems, difficulties, or intention to drop out.
Alarm reminders were delivered four times a day during 60-minute blocks (8–9 am, 12–1 pm, 5–6 pm, and 9–10 pm). The completion rate of time four (the block between 9 and 10 pm) was lower than that for other times, but the difference was not significant (p = 0.072). Nonetheless, this needs to be addressed since not everyone might have been awake during the study hours since participants were instructed to follow their own daily circadian rhythms. Moreover, in the current study, participants used their own smartphones, with which they were very familiar. The reminder beeps thus could easily have been missed since the participants might have been so accustomed to their phones' sounds that they had learned to ignore them at busy times. This would have had consequences for the completion rates, as identified in a previous EMA study using an electronic diary [24].
This study had limitations. The current study included patients who responded to the EMA study, and those who did not respond were excluded although they initially agreed to participate. Another limitation is that reasons for dropping out during the EMA survey period were not evaluated with the exception of a few patients who gave feedback noting they were too busy to actively participate. In-depth individual or focus group interviews are a potentially useful method to assess factors associated with completion rates for future studies.
Considering the known variables related to study design such as EMA duration, frequency, and alarm times, strategic approaches should be established to improve completion rates and the quality of data in EMA studies. Additionally, participant-centered strategies for building positive relationships and communication among participants and researchers must be prepared to motivate and encourage participants to remain involved throughout the study duration of a longitudinal EMA study.
XML Download