Healthc Inform Res Search


Healthc Inform Res > Volume 29(2); 2023 > Article
Rahmadhan, Azizi, Handayani, and Monicha: Design of a Reminder and Recall System in a Contact Tracing Application to Support Coronavirus Booster Vaccination



The rate of coronavirus disease 2019 (COVID-19) booster vaccination in Indonesia remains relatively low, representing 15.33% of the overall vaccination target as of April 2022. The implementation of a reminder and recall system has been shown to be effective in increasing vaccination rates. In prior research, reminders and recalls were sent through traditional media, such as mail, and had not yet been integrated into modern media, such as smartphone applications and (in particular) contact tracing applications. Therefore, the present study was conducted to design a reminder and recall system for the PeduliLindungi contact tracing application.


We used the design science research (DSR) methodology with three iterations. The first iteration produced a low-fidelity prototype (or wireframe), and the next yielded a high-fidelity (clickable) prototype.


The final prototype included three main features: a reminder and recall mechanism, online registration for COVID-19 booster vaccination, and educational articles. The evaluation consisted of interviews in the first iteration, interviews and the System Usability Scale (SUS) questionnaire in the second, and the Post-Study System Usability Questionnaire (PSSUQ) in the third. The SUS value obtained in the second iteration was 71.6, indicating good (acceptable) results, while in the third iteration, the system usefulness, information quality, interface quality, and overall PSSUQ values were 2.456, 2.473, 2.230, and 2.397, respectively, indicating good quality of the resulting design.


This research contributes to two areas: implementation of a reminder and recall system in the PeduliLindungi contact tracing application and enhancement of contact tracing applications using DSR methodology.

I. Introduction

To date, mass vaccination has been shown to effectively control the spread of coronavirus disease 2019 (COVID-19) [1]. In Indonesia, the implementation of COVID-19 vaccination has reached the booster vaccination stage, which began on January 12, 2022 [2]. However, COVID-19 booster vaccination is not mandatory unless one uses specific forms of public transportation, and requirements for booster vaccination are especially limited for non-elderly people [3]. This raises concerns about public acceptance of the COVID-19 booster vaccine in Indonesia. These concerns are supported by the low number of COVID-19 booster vaccinations, which have only reached 15.33% of the total vaccination target as of April 2022 [4]. While the Indonesian government has released a contact tracing application, known as the PeduliLindungi, the application does not yet include a reminder and recall feature regarding booster vaccine eligibility, so users must constantly check the app to see whether they have received an invitation for a booster vaccine.
Reminder and recall systems have been demonstrated effective in increasing vaccination rates among people in various age ranges [510]. The reminder mechanism informs the user about upcoming vaccination dates, while the recall mechanism notifies the user if a scheduled vaccination was missed [11]. Reminder and recall messages can be sent through various media such as telephone calls (manual or automatic), mail, or text messages [12]. Of these media, the most common for reminders and recalls is the telephone call, which may be supplemented by mail [8]. However, the implementation of newer technologies, such as text messages, to send reminders and recalls is promising for several reasons. The main advantage offered by text messaging is the ability to reach a wider audience at a lower cost. In addition, text messages can also reach people who were previously difficult to contact [8]. Moreover, a mobile health system connected to an online registration mechanism is an interesting target for future implementation and research [8].
A reminder and recall system can also be used to encourage adherence to a vaccination schedule, as suggested by previous researchers and vaccination program administrators [1315]. Yunusa et al. [10] found that pediatric patients whose parents received a reminder of three-dose pentavalent vaccination had a higher rate of vaccine completion (59.4%) than patients whose parents received no reminder (34.1%). Hurley et al. [7] also found a reminder and recall system effective in increasing influenza vaccination. In addition, Szilagyi et al. [12] reported that 12 of 15 examined studies indicated that reminder and recall systems can effectively improve adherence to immunization schedules among children.
Although reminder and recall systems have been demonstrated effective for various types of vaccination, no previous studies have discussed the implementation of such a system for COVID-19 and particularly booster vaccination. In addition, no prior studies have covered the integration of reminder and recall systems into smartphone applications in general or contact tracing applications in particular. To date, very few studies have used the design science research (DSR) methodology to develop and improve contact tracing applications. The DSR paradigm is highly relevant for research on mobile health (mHealth) intended to uncover innovative solutions to real-world problems [16]. For example, Behne et al. [17] attempted to improve the German Corona-Warn-App application using DSR with three iterations of user-centered development and evaluation. With the present study, we intended to fill the research gap by developing and integrating a reminder and recall system using the DSR methodology developed by Peffers et al. [18], in an effort to support the implementation of COVID-19 booster vaccination in Indonesia. Hence, the research question of this study was how a reminder and recall system could be effectively designed within Indonesia’s national contact tracing application and used to maximize the implementation of COVID-19 booster vaccination.

II. Methods

The research was conducted using the DSR methodology, which was implemented in three iterations. This methodology is highly relevant and necessary to develop mHealth technology that is not only useful but also safe and effective for users [19]. The DSR methodology consists of stages: 1) problem identification and definition of the objectives for a solution, 2) design and development, 3) artifact demonstration, 4) artifact evaluation, and 6) artifact communication. In the present study, design and development, demonstration, evaluation, and communication were carried out in three iterations. The first iteration was focused on obtaining information, requirements, and features to be implemented as a low-fidelity prototype (wireframe). The second was focused on improving the quality of the designs and converting the wireframe into a high-fidelity (clickable) prototype. The third iteration involved finalizing the design. Figure 1 summarizes the research steps.
The second and third iterations included an online questionnaire to reach more respondents for prototype evaluation. Before distributing the questionnaires to respondents, we conducted a readability test with five respondents to ensure that the questionnaire statements were correct and easily understood. We used the System Usability Scale (SUS) questionnaire in the second iteration and the Post-Study System Usability Questionnaire (PSSUQ) in the third iteration. The SUS questionnaire includes a 5-level Likert scale where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree. The PSSUQ includes a 7-level Likert scale where 1 = strongly agree, 2 = agree, 3 = slightly agree, 4 = neutral, 5 = strongly disagree, 6 = disagree, and 7 = strongly disagree. In addition, in the first and second iterations, respondents were interviewed.

III. Results

1. Problem Identification and Objective Definition

Based on information obtained through media channels, the rate of booster vaccination in Indonesia is still quite low, at 15.33% of the total target as of April 2022 [4]. The PeduliLindungi contact tracing application lacks an online booster registration feature and a mechanism to actively remind users to receive a COVID-19 booster vaccine. In a DSR approach, problem identification is designed to identify the main issues and requirements to be explored. In this study, we included experts at this stage (purposive sampling) because the problem falls within the health information domain, which is relevant to, and understood by, individuals from the health sector and/or users of health applications. Specifically, we conducted problem identification through online interviews with six respondents (age, 22–42 years) who had knowledge in the health sector, including student, alumni, and lecturer at the Faculty of Public Health; health experts (doctors and vaccinators); and a member of the public, regarding the need for a reminder and recall system within the PeduliLindungi application.
The results of the interviews were used to determine which features to include in the application design. The resulting solution was expected to reflect the needs of PeduliLindungi application users and to increase their desire to receive a COVID-19 booster vaccination. We summarized the interview results by categorizing input from respondents into groups based on content analysis (Table 1).

2. First Iteration

1) Design and development

In the first iteration, the design and development stage began with the examination of several applications that incorporate the reminder concept, with the goal of understanding its use in smartphone applications and identifying features to implement. Key features to be implemented in the design included notification messages (reminder and recall), a list of the nearest health facilities, vaccine registration, a vaccine schedule, and articles. The next stage involved making use of case diagrams to identify the user roles and activities involved (Figure 2).
Several types of notification were generated in the first iteration, including booster vaccine invitation, a reminder of booster vaccine registration (education), a reminder of the booster vaccination schedule, a notification of non-attendance of booster vaccination, and notification of the receipt of a booster vaccine certificate. Figure 3 shows a reminder notification user interface created in the first iteration.

2) Demonstration, evaluation, and communication

In this iteration, the demonstration involved presenting the resulting wireframe design. Then, the wireframe was evaluated through interviews with the original six respondents to confirm the ideas incorporated and designs created. The results of the evaluation were then processed using content analysis techniques, with the results shown in Table 2. The communication stage was carried out simultaneously with the evaluation stage to convey and/or publish the results of the application design to stakeholders and related parties, which in this case were people with a background in the health sector as well as potential users (members of the public).

3. Second Iteration

1) Design and development

In the second iteration, design activities were focused on changing the user interface, originally a wireframe, into a clickable prototype. As such, two types of changes were made: feature additions and feature improvements. Figure 4 shows an example of an improved interface.

2) Demonstration, evaluation, and communication

In the second iteration, demonstration involved allowing respondents to run the application prototype design according to provided instructions. Evaluation was conducted using qualitative and quantitative methods. Qualitative evaluation consisted of semi-structured interviews with the six respondents, while quantitative evaluation involved asking respondents to complete a questionnaire designed using the SUS framework. Appendix 1 lists the SUS questions. We used adjective ratings to represent the SUS scores generated in certain categories in the form of qualitative statements, while we used acceptability scores to indicate the level of acceptance of the product based on the associated SUS score. The SUS questionnaire link was distributed through social media channels that are widely used in Indonesia, such as LINE, WhatsApp, Instagram, Twitter, and Facebook, between April 9, 2022 and April 21, 2022. A total of 111 respondents participated, and respondent demographics are shown in Table 3.
In the evaluation stage of the second iteration, the SUS score results fell within the ranges of “good” (score, 71.6) on the adjective ratings scale and “acceptable” on the acceptability scale. These results indicate a strong design, although some aspects should be improved. In addition to its utility in obtaining quantitative data, the SUS questionnaire also contained open questions regarding the application design at that stage. Respondents’ suggestions included statements about typos and better positioning of the vaccine list button on the article detail page.
“Some words are misspelled.” (Respondent ID 90)
“Maybe on the booster vaccine article page, put the register button higher so that it is immediately visible to users without having to scroll down.” (Respondent ID 59)

4. Third Iteration

1) Design and development

In the last iteration, changes were made to the mechanism for sending reminders and recalls based on the evaluation in the previous iteration. One such change was the addition of information to the Registration Denied notification explaining why the registration submitted by the user was rejected by the selected vaccination site. These improvements are shown in Figure 5.

2) Demonstration, evaluation, and communication

In the third iteration, evaluation was conducted using only a quantitative method, via a 16-item questionnaire designed with the PSSUQ framework [20] (Appendix 2). The PSSUQ yields one overall value and three sub-scale factors (system usefulness, information quality, and interface quality) [20]. The PSSUQ online questionnaire link was distributed through social media channels that are widely used in Indonesia, such as LINE, WhatsApp, Instagram, Twitter, and Facebook. The questionnaire was distributed between April 25, 2022 and May 12, 2022, and 113 respondents participated (Table 4). Table 4 shows that most of the respondents were female, students, and aged 15–24 years.
PSSUQ values were obtained by calculating the average of each Likert scale score on each statement, with a lower score indicating greater satisfaction. The results of the PSSUQ data processing are shown in Table 5. The values for the four components indicate a good design, as each fell below the average value for the PSSUQ component. Most participants responded positively to open questions in the PSSUQ questionnaire (qualitative data), as shown below.
“The scenario flow is easy to understand, and the features are also easy to access.” (Respondent ID 68)
“It’s good; I like it because there is information about the booster vaccine, making it easier for someone if they want a booster.” (Respondent ID 85)
“In terms of features, it is very complete; the existing design is also good, simple, and attractive.” (Respondent ID 114)
Hence, the existing application design is considered good in terms of design, ease of use, and completeness of features, and it should greatly assist the community with receiving booster vaccinations.

IV. Discussion

This research involves the novel implementation of a reminder and recall system into an existing COVID-19 contact tracing application. In this study, we used the DSR methodology developed by Peffers et al. [18]. This paradigm is highly relevant to research conducted in the mHealth field to develop innovative solutions [16]. The present study is expected to enrich the theoretical literature on the implementation of reminder and recall in contact tracing applications and on the use of the DSR methodology to develop and improve such applications. We involved health experts (doctors, vaccinators, and lecturers at the Faculty of Public Health), students and alumni of the Faculty of Public Health, and members of the public with knowledge of public health. The reminder and recall mechanism was successfully implemented through the notification feature. Several types of notification that can be categorized as reminder mechanisms for booster vaccination include invitations, registration reminders (education), and schedule reminders. Additionally, the recall category included notification of non-attendance to booster vaccination, which serves to remind users of a vaccination date that has passed.
The results of the PSSUQ analysis of the 113 respondents showed that the design produced in the final iteration had good value both overall and in the system usefulness, information quality, and interface quality components. Moreover, this study indicated that (1) the optimal forms of media for reminders and recalls are application notifications and short message service notifications; (2) reminders and recalls should not be sent continuously over a short time interval, and their intensity should be controllable by the user; (3) a reminder message should be a combination of education and warning; and (4) educational reminders can be sent at multiple levels, starting with explaining, then informing, and then progressing to a warning. Finally, the expected practical benefits include serving as a reference for application service providers, especially PeduliLindungi application developers, and developing a reminder and recall system to support the implementation of COVID-19 booster vaccination.
Previous reminder and recall studies have been conducted by several researchers using traditional media, but similar efforts have not yet been applied to modern media such as smartphone applications. Therefore, this research makes a theoretical contribution through our implementation and integration of a reminder and recall system in a contact tracing application (PeduliLindungi) in Indonesia. The findings also enrich the literature on DSR, especially on contact tracing applications. This is important considering that very little research exists on the development of contact tracing applications using the DSR methodology [17].
This research is also expected to provide practical benefits by serving as a reference for application service providers, especially PeduliLindungi application developers, developing a reminder and recall system to support the implementation of COVID-19 booster vaccination and/or vaccination for other infectious diseases in the future. Developers are expected to know how to implement an effective reminder and recall system and to understand which features should be implemented for infectious diseases in particular.
This study had limitations related to the demographics of the questionnaire respondents, which were concentrated with regard to sex (predominantly female), age range (predominantly 15–24 years), and occupation (predominantly students). Additionally, we did not explore in detail the mechanism for integrating the resulting designs with the PeduliLindungi contact tracing application.
In conclusion, in this study, we designed a prototype with several key features: a reminder and recall mechanism, online registration for COVID-19 booster vaccination, and educational articles. The prototypes had an associated SUS value of 71.6, indicating good (acceptable) usability, while the system usefulness, information quality, interface quality, and overall measures of the PSSUQ were 2.456, 2.473, 2.230, and 2.397, respectively, indicating a well-made design. In the future, this study can be augmented by designing and implementing an integrated immunization information system.


Conflict of Interest

No potential conflict of interest relevant to this article was reported.


We want to convey our gratitude to the Ministry of Education, Culture, Research and Technology for the Grant Higher Education Excellence Basic Research Program (Penelitian Dasar Unggulan Perguruan Tinggi [PDUPT]; No. NKB/UN2. RST/HKP.05.00/2023).

Figure 1
Summary of research steps.
Figure 2
Use case diagram.
Figure 3
First iteration of the reminder notification user interface.
Figure 4
Second iteration of the reminder notification user interface.
Figure 5
Third iteration of the reminder notification user interface.
Table 1
Content analysis results of problem identification
Category Issues Proposed feature
Vaccination information
  • -Lack of vaccination socialization

  • -The emergence of anti-vaccine groups due to lack of information

  • -No centralized source of information

  • -Article features contain actual and comprehensive information

  • -Article feature, reminder feature via app and SMS

Vaccination implementation
  • -Public awareness of the COVID-19 booster vaccination is still very lacking.

  • -The registration mechanism is less effective and creates crowds.

  • -Article feature, reminder feature and vaccination schedule feature

  • -Vaccination schedule feature, vaccine registration feature, nearest health facility list feature

Vaccination apps
  • -Mistakes in data on existing vaccination applications.

  • -Existing applications do not actively provide information and are difficult to use.

  • -It is necessary to enhance the PeduliLindungi application.

  • -App fixes, reminder feature

COVID-19: coronavirus disease 2019, SMS: short message service.

Table 2
Content analysis results of evaluation in the first iteration
Design Issue
Notification of not attending booster vaccination The first reminder notification is sent on D+1. For the second and third is sent with an interval of 7 days.
Vaccination online registration Things to prepare before vaccination
Sets a limit on the days the user can choose
Vaccination process Add a feedback feature for vaccination providers
Booster vaccination invitation notifications Add a scenario if a user is detected positive when getting a booster vaccination invitation
Booster vaccination registration reminder notifications (education) Add a way to access educational articles without going through notifications
Eye-catching reminder messages on educational notifications
Notification history Separate check-in notifications and educational notifications
Table 3
Summary of System Usability Scale respondent demographics
Demographics Number of respondents (%)
 Women 62 (56)
 Men 49 (44)

Age (yr)
 15–24 100 (90)
 25–34 6 (5)
 35–44 3 (3)
 45–54 2 (2)

 Students 91 (82)
 Entrepreneurs 8 (7)
 Others 12 (11)
Table 4
Summary of Post-Study System Usability Questionnaire respondent demographics
Demographics Number of respondents (%)
 Women 63 (56)
 Men 50 (44)

Age (yr)
 15–24 99 (88)
 25–34 6 (5)
 35–44 3 (3)
 45–54 5 (4)

 Students 87 (77)
 Entrepreneurs 8 (7)
 Others 18 (16)
Table 5
Summary of Post-Study System Usability Questionnaire scores
Object assessment Application design (third iteration)
System usefulness 2.456
Information quality 2.473
Interface quality 2.230
Overall 2.397


1. Lau CS. Can COVID-19 vaccines stop the pandemic? Hong Kong Med J 2021 27(2):84-5.
crossref pmid
2. Rokom . The Ministry of Health issues a circular on the implementation of booster vaccinations in all regions [Internet]. Jakarta, Indonesia: Sehat Negeriku; 2022 [cited at 2023 Mar 10]. Available from:

3. Saptohutomo AP. Many are confused, is Covid-19 booster vaccination mandatory or not? [Internet]. Jakarta, Indonesia: Kontan News; 2022 [cited at 2023 Mar 10]. Available from:

4. Ministry of Health, Republic of Indonesia. National COVID-19 vaccination [Internet]. Jakarta, Indonesia: Ministry of Health, Republic of Indonesia; 2022 [cited at 2023 Mar 10]. Available from:

5. Coley S, Hoefer D, Rausch-Phung E. A population-based reminder intervention to improve human papillomavirus vaccination rates among adolescents at routine vaccination age. Vaccine 2018 36(32 Pt B):4904-9.
crossref pmid
6. Dombkowski KJ, Cowan AE, Reeves SL, Foley MR, Dempsey AF. The impacts of email reminder/recall on adolescent influenza vaccination. Vaccine 2017 35(23):3089-95.
crossref pmid
7. Hurley LP, Beaty B, Lockhart S, Gurfinkel D, Breslin K, Dickinson M, et al. RCT of centralized vaccine reminder/recall for adults. Am J Prev Med 2018 55(2):231-9.
crossref pmid
8. Jong KM, Sikora CA, MacDonald SE. Childhood immunization appointment reminders and recalls: strengths, weaknesses and opportunities to increase vaccine coverage. Public Health 2021 194:170-5.
crossref pmid
9. Szilagyi PG, Albertin CS, Saville AW, Valderrama R, Breck A, Helmkamp L, et al. Effect of state immunization information system based reminder/recall for influenza vaccinations: a randomized trial of autodialer, text, and mailed messages. J Pediatr 2020 221:123-31e4.
crossref pmid pmc
10. Yunusa U, Ibrahim AH, Ladan MA, Gomaa HE. Effect of mobile phone text message and call reminders in the completeness of pentavalent vaccines in Kano state, Nigeria. J Pediatr Nurs 2022 64:e77-e83.
crossref pmid
11. Centers for Disease Control and Prevention (CDC). Recommendations of the Advisory Committee on Immunization Practices, the American Academy of Pediatrics, and the American Academy of Family Physicians: use of reminder and recall by vaccination providers to increase vaccination rates. MMWR Morb Mortal Wkly Rep 1998;47(34):715-7.
crossref pmid
12. Szilagyi PG, Bordley C, Vann JC, Chelminski A, Kraus RM, Margolis PA, et al. Effect of patient reminder/recall interventions on immunization rates: a review. JAMA 2000 284(14):1820-7.
crossref pmid
13. Bell CA, Simmonds KA, MacDonald SE. Exploring the heterogeneity among partially vaccinated children in a population-based cohort. Vaccine 2015 33(36):4572-8.
crossref pmid
14. Lemstra M, Rajakumar D, Thompson A, Moraros J. The effectiveness of telephone reminders and home visits to improve measles, mumps and rubella immunization coverage rates in children. Paediatr Child Health 2011 16(1):e1-5.
crossref pmid pmc
15. Thomas M, Kohli V, King D. Barriers to childhood immunization: findings from a needs assessment study. Home Health Care Serv Q 2004 23(2):19-39.
16. Schnall R, Rojas M, Travers J, Brown W, Bakken S. Use of design science for informing the development of a mobile app for persons living with HIV. AMIA Annu Symp Proc 2014;2014:1037-45.
pmid pmc
17. Behne A, Kruger N, Beinke JH, Teuteberg F. Learnings from the design and acceptance of the German COVID-19 tracing app for IS-driven crisis management: a design science research. BMC Med Inform Decis Mak 2021 21(1):238.
crossref pmid pmc
18. Peffers K, Tuunanen T, Rothenberger MA, Chatterjee S. A design science research methodology for information systems research. J Manag Inf Syst 2007 24(3):45-77.
19. Cross N. Designerly ways of knowing: design discipline versus design science. Des Issues 2001;17(3):49-55.
20. Fruhling A, Lee S. Assessing the reliability, validity and adaptability of PSSUQ. Proceedings of the 11th Americas Conference on Information Systems (AMCIS); 2005 Aug 11–14. Omaha, NE.


Appendix 1

System Usability Scale questionnaire

Code Questionnaire item
S1 I think I will use this application often
S2 I think this application is too complex (displays many unnecessary things)
S3 I find this application easy to use
S4 I need technical assistance (tutorial/FAQ) to use this application
S5 I think the features provided in this application are well designed
S6 I judge too many inconsistencies in this application (example: features, images, explanations)
S7 I feel most people will quickly find it easy to use this application
S8 I think this application flow is very complicated to use
S9 I feel very confident using this application
S10 I need to learn many things to use this application properly
Appendix 2

Post-Study System Usability Questionnaire

Dimension Description Code Questionnaire statement
System usefulness A system that can be easily learned and used, can effectively complete its tasks, and can quickly increase the productivity of its users. P1 Overall, I am satisfied with the ease of use of the application.
P2 I find it easy to use this application.
P3 I can complete tasks and scenarios fast through this application.
P4 I feel comfortable using this application.
P5 I find it easy to learn how to use this application.
P6 I will be productive using this application.

Information quality The information provided by the system can be easily understood, can effectively assist in task completion and organization. P7 I got an error message explaining how to fix it.
P8 Whenever I make a mistake in the application, I can easily and quickly fix it.
P9 I feel the help information provided by the application is very clear.
P10 I find it easy to find the information needed.
P11 The information provided by the app was effective in helping me work on assignments and scenarios.
P12 The information arrangement provided by the application is quite clear.

Interface quality The system created provides satisfaction to its users both in appearance and as a whole P13 I feel the interface is quite comfortable.
P14 I like the interface in the application.
P15 I feel that all the functions and capabilities provided by the application have met my expectations.
P16 Overall, I am satisfied with this application.
Share :
Facebook Twitter Linked In Google+ Line it
METRICS Graph View
  • 0 Crossref
  •     Scopus
  • 436 View
  • 56 Download
Related articles in Healthc Inform Res


Browse all articles >

Editorial Office
1618 Kyungheegung Achim Bldg 3, 34, Sajik-ro 8-gil, Jongno-gu, Seoul 03174, Korea
Tel: +82-2-733-7637, +82-2-734-7637    E-mail:                

Copyright © 2023 by Korean Society of Medical Informatics.

Developed in M2community

Close layer
prev next