PDF
Citation
Print
I. Introduction
To date, mass vaccination has been shown to effectively control the spread of coronavirus disease 2019 (COVID-19) [1]. In Indonesia, the implementation of COVID-19 vaccination has reached the booster vaccination stage, which began on January 12, 2022 [2]. However, COVID-19 booster vaccination is not mandatory unless one uses specific forms of public transportation, and requirements for booster vaccination are especially limited for non-elderly people [3]. This raises concerns about public acceptance of the COVID-19 booster vaccine in Indonesia. These concerns are supported by the low number of COVID-19 booster vaccinations, which have only reached 15.33% of the total vaccination target as of April 2022 [4]. While the Indonesian government has released a contact tracing application, known as the PeduliLindungi, the application does not yet include a reminder and recall feature regarding booster vaccine eligibility, so users must constantly check the app to see whether they have received an invitation for a booster vaccine.
Reminder and recall systems have been demonstrated effective in increasing vaccination rates among people in various age ranges [5–10]. The reminder mechanism informs the user about upcoming vaccination dates, while the recall mechanism notifies the user if a scheduled vaccination was missed [11]. Reminder and recall messages can be sent through various media such as telephone calls (manual or automatic), mail, or text messages [12]. Of these media, the most common for reminders and recalls is the telephone call, which may be supplemented by mail [8]. However, the implementation of newer technologies, such as text messages, to send reminders and recalls is promising for several reasons. The main advantage offered by text messaging is the ability to reach a wider audience at a lower cost. In addition, text messages can also reach people who were previously difficult to contact [8]. Moreover, a mobile health system connected to an online registration mechanism is an interesting target for future implementation and research [8].
A reminder and recall system can also be used to encourage adherence to a vaccination schedule, as suggested by previous researchers and vaccination program administrators [13–15]. Yunusa et al. [10] found that pediatric patients whose parents received a reminder of three-dose pentavalent vaccination had a higher rate of vaccine completion (59.4%) than patients whose parents received no reminder (34.1%). Hurley et al. [7] also found a reminder and recall system effective in increasing influenza vaccination. In addition, Szilagyi et al. [12] reported that 12 of 15 examined studies indicated that reminder and recall systems can effectively improve adherence to immunization schedules among children.
Although reminder and recall systems have been demonstrated effective for various types of vaccination, no previous studies have discussed the implementation of such a system for COVID-19 and particularly booster vaccination. In addition, no prior studies have covered the integration of reminder and recall systems into smartphone applications in general or contact tracing applications in particular. To date, very few studies have used the design science research (DSR) methodology to develop and improve contact tracing applications. The DSR paradigm is highly relevant for research on mobile health (mHealth) intended to uncover innovative solutions to real-world problems [16]. For example, Behne et al. [17] attempted to improve the German Corona-Warn-App application using DSR with three iterations of user-centered development and evaluation. With the present study, we intended to fill the research gap by developing and integrating a reminder and recall system using the DSR methodology developed by Peffers et al. [18], in an effort to support the implementation of COVID-19 booster vaccination in Indonesia. Hence, the research question of this study was how a reminder and recall system could be effectively designed within Indonesia’s national contact tracing application and used to maximize the implementation of COVID-19 booster vaccination.
II. Methods
The research was conducted using the DSR methodology, which was implemented in three iterations. This methodology is highly relevant and necessary to develop mHealth technology that is not only useful but also safe and effective for users [19]. The DSR methodology consists of stages: 1) problem identification and definition of the objectives for a solution, 2) design and development, 3) artifact demonstration, 4) artifact evaluation, and 6) artifact communication. In the present study, design and development, demonstration, evaluation, and communication were carried out in three iterations. The first iteration was focused on obtaining information, requirements, and features to be implemented as a low-fidelity prototype (wireframe). The second was focused on improving the quality of the designs and converting the wireframe into a high-fidelity (clickable) prototype. The third iteration involved finalizing the design. Figure 1 summarizes the research steps.
The second and third iterations included an online questionnaire to reach more respondents for prototype evaluation. Before distributing the questionnaires to respondents, we conducted a readability test with five respondents to ensure that the questionnaire statements were correct and easily understood. We used the System Usability Scale (SUS) questionnaire in the second iteration and the Post-Study System Usability Questionnaire (PSSUQ) in the third iteration. The SUS questionnaire includes a 5-level Likert scale where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree. The PSSUQ includes a 7-level Likert scale where 1 = strongly agree, 2 = agree, 3 = slightly agree, 4 = neutral, 5 = strongly disagree, 6 = disagree, and 7 = strongly disagree. In addition, in the first and second iterations, respondents were interviewed.
III. Results
1. Problem Identification and Objective Definition
Based on information obtained through media channels, the rate of booster vaccination in Indonesia is still quite low, at 15.33% of the total target as of April 2022 [4]. The PeduliLindungi contact tracing application lacks an online booster registration feature and a mechanism to actively remind users to receive a COVID-19 booster vaccine. In a DSR approach, problem identification is designed to identify the main issues and requirements to be explored. In this study, we included experts at this stage (purposive sampling) because the problem falls within the health information domain, which is relevant to, and understood by, individuals from the health sector and/or users of health applications. Specifically, we conducted problem identification through online interviews with six respondents (age, 22–42 years) who had knowledge in the health sector, including student, alumni, and lecturer at the Faculty of Public Health; health experts (doctors and vaccinators); and a member of the public, regarding the need for a reminder and recall system within the PeduliLindungi application.
The results of the interviews were used to determine which features to include in the application design. The resulting solution was expected to reflect the needs of PeduliLindungi application users and to increase their desire to receive a COVID-19 booster vaccination. We summarized the interview results by categorizing input from respondents into groups based on content analysis (Table 1).
2. First Iteration
1) Design and development
In the first iteration, the design and development stage began with the examination of several applications that incorporate the reminder concept, with the goal of understanding its use in smartphone applications and identifying features to implement. Key features to be implemented in the design included notification messages (reminder and recall), a list of the nearest health facilities, vaccine registration, a vaccine schedule, and articles. The next stage involved making use of case diagrams to identify the user roles and activities involved (Figure 2).
Several types of notification were generated in the first iteration, including booster vaccine invitation, a reminder of booster vaccine registration (education), a reminder of the booster vaccination schedule, a notification of non-attendance of booster vaccination, and notification of the receipt of a booster vaccine certificate. Figure 3 shows a reminder notification user interface created in the first iteration.
2) Demonstration, evaluation, and communication
In this iteration, the demonstration involved presenting the resulting wireframe design. Then, the wireframe was evaluated through interviews with the original six respondents to confirm the ideas incorporated and designs created. The results of the evaluation were then processed using content analysis techniques, with the results shown in Table 2. The communication stage was carried out simultaneously with the evaluation stage to convey and/or publish the results of the application design to stakeholders and related parties, which in this case were people with a background in the health sector as well as potential users (members of the public).
3. Second Iteration
1) Design and development
In the second iteration, design activities were focused on changing the user interface, originally a wireframe, into a clickable prototype. As such, two types of changes were made: feature additions and feature improvements. Figure 4 shows an example of an improved interface.
2) Demonstration, evaluation, and communication
In the second iteration, demonstration involved allowing respondents to run the application prototype design according to provided instructions. Evaluation was conducted using qualitative and quantitative methods. Qualitative evaluation consisted of semi-structured interviews with the six respondents, while quantitative evaluation involved asking respondents to complete a questionnaire designed using the SUS framework. Appendix 1 lists the SUS questions. We used adjective ratings to represent the SUS scores generated in certain categories in the form of qualitative statements, while we used acceptability scores to indicate the level of acceptance of the product based on the associated SUS score. The SUS questionnaire link was distributed through social media channels that are widely used in Indonesia, such as LINE, WhatsApp, Instagram, Twitter, and Facebook, between April 9, 2022 and April 21, 2022. A total of 111 respondents participated, and respondent demographics are shown in Table 3.
In the evaluation stage of the second iteration, the SUS score results fell within the ranges of “good” (score, 71.6) on the adjective ratings scale and “acceptable” on the acceptability scale. These results indicate a strong design, although some aspects should be improved. In addition to its utility in obtaining quantitative data, the SUS questionnaire also contained open questions regarding the application design at that stage. Respondents’ suggestions included statements about typos and better positioning of the vaccine list button on the article detail page.
“Some words are misspelled.” (Respondent ID 90)
“Maybe on the booster vaccine article page, put the register button higher so that it is immediately visible to users without having to scroll down.” (Respondent ID 59)
4. Third Iteration
1) Design and development
In the last iteration, changes were made to the mechanism for sending reminders and recalls based on the evaluation in the previous iteration. One such change was the addition of information to the Registration Denied notification explaining why the registration submitted by the user was rejected by the selected vaccination site. These improvements are shown in Figure 5.
2) Demonstration, evaluation, and communication
In the third iteration, evaluation was conducted using only a quantitative method, via a 16-item questionnaire designed with the PSSUQ framework [20] (Appendix 2). The PSSUQ yields one overall value and three sub-scale factors (system usefulness, information quality, and interface quality) [20]. The PSSUQ online questionnaire link was distributed through social media channels that are widely used in Indonesia, such as LINE, WhatsApp, Instagram, Twitter, and Facebook. The questionnaire was distributed between April 25, 2022 and May 12, 2022, and 113 respondents participated (Table 4). Table 4 shows that most of the respondents were female, students, and aged 15–24 years.
PSSUQ values were obtained by calculating the average of each Likert scale score on each statement, with a lower score indicating greater satisfaction. The results of the PSSUQ data processing are shown in Table 5. The values for the four components indicate a good design, as each fell below the average value for the PSSUQ component. Most participants responded positively to open questions in the PSSUQ questionnaire (qualitative data), as shown below.
“The scenario flow is easy to understand, and the features are also easy to access.” (Respondent ID 68)
“It’s good; I like it because there is information about the booster vaccine, making it easier for someone if they want a booster.” (Respondent ID 85)
“In terms of features, it is very complete; the existing design is also good, simple, and attractive.” (Respondent ID 114)
Hence, the existing application design is considered good in terms of design, ease of use, and completeness of features, and it should greatly assist the community with receiving booster vaccinations.
IV. Discussion
This research involves the novel implementation of a reminder and recall system into an existing COVID-19 contact tracing application. In this study, we used the DSR methodology developed by Peffers et al. [18]. This paradigm is highly relevant to research conducted in the mHealth field to develop innovative solutions [16]. The present study is expected to enrich the theoretical literature on the implementation of reminder and recall in contact tracing applications and on the use of the DSR methodology to develop and improve such applications. We involved health experts (doctors, vaccinators, and lecturers at the Faculty of Public Health), students and alumni of the Faculty of Public Health, and members of the public with knowledge of public health. The reminder and recall mechanism was successfully implemented through the notification feature. Several types of notification that can be categorized as reminder mechanisms for booster vaccination include invitations, registration reminders (education), and schedule reminders. Additionally, the recall category included notification of non-attendance to booster vaccination, which serves to remind users of a vaccination date that has passed.
The results of the PSSUQ analysis of the 113 respondents showed that the design produced in the final iteration had good value both overall and in the system usefulness, information quality, and interface quality components. Moreover, this study indicated that (1) the optimal forms of media for reminders and recalls are application notifications and short message service notifications; (2) reminders and recalls should not be sent continuously over a short time interval, and their intensity should be controllable by the user; (3) a reminder message should be a combination of education and warning; and (4) educational reminders can be sent at multiple levels, starting with explaining, then informing, and then progressing to a warning. Finally, the expected practical benefits include serving as a reference for application service providers, especially PeduliLindungi application developers, and developing a reminder and recall system to support the implementation of COVID-19 booster vaccination.
Previous reminder and recall studies have been conducted by several researchers using traditional media, but similar efforts have not yet been applied to modern media such as smartphone applications. Therefore, this research makes a theoretical contribution through our implementation and integration of a reminder and recall system in a contact tracing application (PeduliLindungi) in Indonesia. The findings also enrich the literature on DSR, especially on contact tracing applications. This is important considering that very little research exists on the development of contact tracing applications using the DSR methodology [17].
This research is also expected to provide practical benefits by serving as a reference for application service providers, especially PeduliLindungi application developers, developing a reminder and recall system to support the implementation of COVID-19 booster vaccination and/or vaccination for other infectious diseases in the future. Developers are expected to know how to implement an effective reminder and recall system and to understand which features should be implemented for infectious diseases in particular.
This study had limitations related to the demographics of the questionnaire respondents, which were concentrated with regard to sex (predominantly female), age range (predominantly 15–24 years), and occupation (predominantly students). Additionally, we did not explore in detail the mechanism for integrating the resulting designs with the PeduliLindungi contact tracing application.
In conclusion, in this study, we designed a prototype with several key features: a reminder and recall mechanism, online registration for COVID-19 booster vaccination, and educational articles. The prototypes had an associated SUS value of 71.6, indicating good (acceptable) usability, while the system usefulness, information quality, interface quality, and overall measures of the PSSUQ were 2.456, 2.473, 2.230, and 2.397, respectively, indicating a well-made design. In the future, this study can be augmented by designing and implementing an integrated immunization information system.
XML Download