I. Introduction
Access to essential medicines in developing countries is a critical global health issue [1]. Many communicable diseases, which lead to death and disability in these regions, can be prevented, treated, or mitigated with the availability of cost-effective essential medicines, including vaccines [2,3]. Immunization not only saves lives but also shields against serious illnesses and is acknowledged as one of the most cost-effective health interventions [4]. The World Health Organization (WHO) has identified equitable access to essential medicines, vaccines, and technologies as a fundamental component of any national health system [5].
However, one-third of the global population, including half of those in Sub-Saharan countries, lacks regular access to life-saving essential medicines and vaccines [1,6,7]. Although economic constraints in developing countries play a role, structural and health system-related factors are also significant bottlenecks in the supply chain for essential medicines and vaccines [8].
The primary purpose of the healthcare supply chain is to ensure that the right product is delivered to the right customer, in the correct quantity and condition, to the appropriate location, at the right time, and at an affordable cost [9]. Despite the complexity of the healthcare supply chain, meeting all these criteria is essential for the effective functioning of a healthcare system [8,9].
An efficient vaccine supply chain system is crucial for achieving the goals of the Expanded Program for Immunization (EPI), a key intervention in reducing mortality and morbidity among under-5 children [4,10]. The success of the EPI heavily relies on a specialized supply chain management system for vaccines, which includes proper storage, transportation, handling, and the involvement of qualified personnel [4,11]. Given that vaccines are temperature-sensitive biological substances, a robust end-to-end cold chain system is essential to maintain vaccine potency up to the point of delivery [11]. Without this, vaccines can rapidly lose their effectiveness or even become hazardous [12].
A robust vaccine logistics management information system (VLMIS) is essential for managing supplies, predicting demand, and strategic planning [13]. According to the WHO, an effective health information system that supports the production, analysis, and dissemination of data is crucial for the proper functioning of a healthcare system [5].
Likewise, an effective VLMIS, which provides real-time logistics data at every healthcare level, is crucial for a resilient and successful immunization program [9,11,14,15]. Its primary goal is to ensure an uninterrupted supply and optimal use of high-quality vaccines from manufacturers to service delivery points [9,16].
In Ethiopia, the EPI was launched in 1980, initially providing six antigens [10]. Starting in June 2014, the vaccine supply chain began transitioning to a phased approach, integrating with the Ethiopian Pharmaceutical Supply Service (EPSS) Integrated Pharmaceutical Logistics System (IPLS). Furthermore, the VLMIS has been incorporated into the IPLS using adapted logistics management information system (LMIS) tools [15,17].
Despite these efforts, studies in Ethiopia have identified numerous challenges at health facilities concerning immunization data quality. These issues encompass the use of non-standardized data recording and reporting tools, unreliable data, shortages of recording materials, poor data quality, and limited capacity among EPI officers [10,18–21]. Similar challenges have been documented in other low- and middle-income countries [22]. Although vaccine logistics management is a critical component of the EPI program, there has been a lack of in-depth research on the VLMIS and related data quality issues. Therefore, evaluating the performance of the existing VLMIS is crucial for enhancing the vaccine supply chain system in Ethiopia’s public health facilities.
II. Methods
1. Study Design
An institution-based concurrent explanatory mixed study design was implemented in Amhara Region, the second-most populous region in Ethiopia. Data collection involved a combination of quantitative and qualitative methods.
2. Study Population
The study population comprised hospitals and health centers that provide EPI services, along with all EPI focal persons responsible for managing vaccines. Both manual and digital VLMIS implementations in the region were included in the evaluation.
3. Sampling Procedure
Six zonal and two city administrations were selected and stratified into three clusters according to the locations of EPSS hubs. Fifteen percent of the facilities were included following the recommendation from USAID’s Logistics Indicators Assessment Tool [23]. A multi-stage stratified random sampling approach was used, with sample sizes for each zone and district allocated proportionally. In total, 102 facilities participated, comprising referral hospitals, general hospitals, primary hospitals, and health centers.
4. Inclusion and Exclusion Criteria
Inclusion criteria are all zonal administrations; public health facilities that had been functional for more than 1 year; EPI focal persons with >6 months of experience for the qualitative interview; and manual/digital VLMIS implemented in more than one zonal administration.
Exclusion criteria are health facilities that had been damaged in conflict-affected areas or had security problems during the study; manual and digital information management tools for other health services; and health posts.
5. Data Collection Process
The study utilized a structured data abstraction checklist and interviewer-administered questionnaires to collect quantitative data, while qualitative data were gathered using guiding questions. The tools for data collection were adapted from those developed by the WHO, the Ministry of Health, and the Amhara Regional Health Bureau. Data were extracted from vaccine ledger books, vaccine request form (VRF) reports, and VLMIS forms. The interviewer-administered questionnaires gathered information from EPI focal points about their experiences and the performance of the VLMIS at their facilities. Additionally, observations and physical counts of selected vaccines in refrigerators were conducted to verify inventory accuracy. Qualitative information was collected from the EPI focal points through guiding questions.
6. Data Quality Assurance
A triangulation technique was employed to ensure the validity of the data. Data collection tools, adapted from those developed by the WHO and the Federal Ministry of Health, were validated by 10 field experts. These tools were pre-tested, and necessary amendments were made prior to the commencement of data collection. Qualified health supply chain management postgraduate students, who had received two days of training, conducted the data collection. Field supervisors checked the completeness of the data on a daily basis.
7. Data Analysis and Interpretation
Quantitative data were thoroughly checked for completeness and consistency before being entered into Epi Info software. After validation, the data were exported to SPSS version 23.0 (IBM Corp., Armonk, NY, USA) for detailed descriptive statistical analysis. The results were subsequently presented through graphs and tables and interpreted in line with the study objectives. Qualitative data, which captured the experiences of EPI focal persons regarding VLMIS performance, were subjected to thematic analysis. This analysis involved translating audio recordings and manually coding the data. Key themes and patterns were then identified and interpreted to gain a comprehensive understanding of the VLMIS from the users’ perspectives.
8. Ethics Statement
This study was approved by the Ethical Review Committee of the School of Pharmacy (Protocol No. ERB/SOP/399/14/2022) and the Institutional Review Board of College of Health Sciences, Addis Ababa University (Protocol No. 007/22/SoP). Informed consent was obtained from all participants involved in the study.
III. Results
1. General Information
VLMIS data were collected from 99 of the 102 public health facilities included in the study. Two facilities had not initiated EPI services, and one had its operations disrupted due to conversion into a COVID-19 treatment center. Of the facilities surveyed, 37 (37.4%) were located in urban areas, with the remainder in rural settings. The majority of the facilities (n = 84; 84.8%), were health centers. The remaining facilities were primary hospitals, general hospitals, and referral hospitals, which accounted for nine (9.1%), three (3%), and three (3%) respectively (Figure 1).
Among the facilities surveyed, 68 (68.7%) lacked a separate vaccine storage room and instead used shared spaces with immunization service areas. All but one facility had an assigned EPI focal person. Manual VLMIS was utilized in all facilities, with no instances of digital VLMIS adoption.
All surveyed facilities were equipped with at least one functional refrigerator. During the evaluation, the average availability of functional temperature monitoring devices within these refrigerators was 93.9% (standard deviation ±17.8). Of these devices, temperature recording charts were present in 89.5% of cases (standard deviation ±23.1). Over the course of a month, only 82 facilities (82.8%) consistently recorded refrigerator temperatures in accordance with WHO recommendations.
2. Availability and Utilization of VLMIS Forms
During the study, the average availability of 15 VLMIS forms was 44.1%, with a utilization rate of 40.5%. The VRF, which serves as a monthly consumption report and order request, was present in 96 (97%) of the surveyed facilities. In contrast, the bin card and stock card, both used for stock-keeping records, were found in only one or no facilities. The VRF maintained a better 3-month supply rate (93.9%), whereas the vaccine accuracy analysis tool form was in an exceptionally deficient supply (5.1%) (Table 1).
3. Data Quality
VLMIS data quality was evaluated across four dimensions: inventory accuracy rate, timeliness, completeness, and accuracy. The first dimension evaluated the accuracy of stock-keeping records, whereas the other three dimensions concentrated on VRF reports. For inventory accuracy, three key tracer vaccines used in child care and routine immunization were analyzed, while all eight tracer vaccines were examined for the remaining data quality dimensions.
1) Inventory accuracy rate for vaccines
Out of the 99 facilities providing vaccination services, only 84 (84.8%) utilized the vaccine stock management ledger book, and merely 42 (50%) of those updated it prior to the day of the visit. The mean inventory discrepancy rate for the three tracer vaccines—pentavalent, Bacillus Calmette-Guerin (BCG), and measles—was 15.9 ± 42.1 (range 0–255.2), 26.5 ± 59.6 (range 0–350), and 25 ± 66.3 (range 0–400), respectively. The overall mean inventory accuracy rate for these tracer vaccines stood at 58.3% ± 45.1%.
2) Timeliness of VRF reports
Public health facilities are required to submit monthly vaccine consumption and demand requests using the VRF. Woreda health offices or EPSS branches then distribute vaccines based on these reports. This study analyzed one year of monthly VRF reports to evaluate their timeliness. In the most recent reporting period, 85 facilities (85.9%) correctly indicated the reporting date, while 13 facilities (13.1%) failed to submit a report. Alarmingly, 44 facilities (44.4%) did not have a VRF report for the 12th reporting period. Approximately 65 facilities (65.7%) submitted VRF reports on time in the recent period, but this number dropped to 41 facilities (41.4%) in the latest period. The data indicate a declining trend in the timeliness of VRF report submissions over time (Figure 2).
3) Completeness of VRF reports
Public health facilities utilize distinct versions of the VRF for urban and rural settings; however, these forms must be used more clearly and consistently. Three rural facilities mistakenly used the urban VRF, while 12 urban facilities utilized the rural VRF. The completeness of the data for both urban and rural facilities was evaluated based on the VRF employed during the assessment.
4) Urban VRF report completeness
Of the 28 health facilities using the urban VRF, 26 submitted recent VRF reports, achieving an overall completeness rate of 88.4%. The data element “quantity of vaccines received” had the highest completeness score at 95.6%, whereas the “loss/adjustment” element recorded the lowest score at 76.1% (Table 2).
5) Rural VRF report completeness
Of the 71 health facilities utilizing the rural VRF, 60 provided recent reports, achieving an overall completeness rate of 87.8%. The data element “beginning balance” recorded the highest completeness at 98.3%. In contrast, the completeness scores for “doses discarded” and “vaccinations given in the last supply period” were the lowest, at 79.8% and 40.8%, respectively (Table 3).
6) Accuracy of VRF reports
The accuracy of both the urban and rural versions of the VRF was assessed.
7) Urban VRF report accuracy
The accuracy of VRF reports was assessed using only the most recent data available. The average accuracy across various facilities was 73.4%. The “quantity received” category achieved the highest accuracy, scoring 90.6%. In contrast, the “quantity needed (requested amount)” and “maximum stock” categories recorded the lowest accuracy scores, at 62.5% and 58.3%, respectively. The average accuracy rate for all eight vaccines across health facilities did not exceed 75% (Table 4).
8) Rural VRF report accuracy
The average accuracy rate of VRF reports for all eight vaccines in health facilities using rural VRF was 69.6%. The highest accuracy score (89.4%) was for “receiving during the last supply period,” while the lowest score (31.3%) was for “vaccinations given in the last supply period” (Table 5).
4. EPI Focal Persons’ Experiences and Reflections on VLMIS Performance
Out of the 99 facilities evaluated for VLMIS performance, 98 had EPI focal persons overseeing the vaccine supply chain. We interviewed 96 EPI focal persons, who had an average age of 28.97 ± 4.21 years. The majority, 51 (53.1%), were female nurses. In terms of academic qualifications, 36 (37.5%) held bachelor’s degrees, while 60 (62.5%) possessed diplomas. On average, these individuals had 6.9±4.03 years of professional experience, including 2.1 ± 1.54 years as EPI focal persons. Their experience in other pharmaceutical supply chain-related activities averaged 0.3 ± 0.87 years.
1) Experiences of EPI focal persons related to VLMIS performance
(1) Vaccine logistics training
Of the EPI focal persons surveyed, 83 (86.5%) reported having participated in training related to vaccine and cold chain logistics management. The most frequently mentioned training topics were “Effective vaccine management,” “EPI vaccine management,” and “Vaccine cold chain management.” These topics were cited by 37 (44.6%), 28 (33.7%), and 25 (30.1%) of the trained EPI focal persons, respectively (Figure 3).
(2) VLMIS supportive supervision
Among the EPI focal persons surveyed, 73 (76%) reported receiving supportive supervision for vaccine supply chain management and VLMIS-related issues. Of these, 25 (34.2%) received supervision within the last month, while 27 (37.0%) received it 1–3 months ago, 11 (15.1%) 3–6 months ago, and 10 (13.7%) more than 6 months ago. The primary organizations providing supportive supervision were the Woreda health office and the zonal health department, cited by 30 (41.1%) and 15 (20.5%) respondents, respectively. Additionally, joint teams from various organizations and NGOs were mentioned by 15 (20.5%) respondents, and another 11 (15.1%) cited other sources (Figure 4).
Supportive supervision primarily concentrated on various tasks including recording in vaccine ledger books, checking vaccine vial monitors, monitoring temperatures, managing cold storage, overseeing immunization, reporting through the VRF, preparing micro plans, ensuring data quality, recording vaccine wastage, forecasting, and maintaining the availability of LMIS forms. EPI focal persons reported improvements in these areas following supportive supervision.
(3) VLMIS feedback
Of the 96 EPI focal persons interviewed, only 33 (34.4%) received written feedback from higher institutions last year. Among these, 24 (72.7%) received their feedback from the Woreda health office, while the remainder received feedback from the zonal health department and the regional health bureau. Of these individuals, only 19 (57.6%) received written feedback in the most recent quarter (3 months).
(4) Challenges and suggestions to improve the VLMIS
EPI focal persons shared challenges and recommendations for improving the VLMIS in health facilities, which are summarized thematically below (Table 6).
IV. Discussion
This survey evaluated the status of the VLMIS in Amhara Region, Ethiopia. A specialized, well-equipped infrastructure is essential for managing the vaccine supply chain [11]. In this study, only 68 (68.7%) of the surveyed health facilities had dedicated vaccine storage rooms. Additionally, only 93.9% (standard deviation ±17.8) of the functional refrigerators were equipped with functional temperature monitoring devices. The main infrastructural challenges identified in these facilities included poor road infrastructure, lack of power backup, and insufficient information and communication technology (ICT) materials. Responses from EPI focal persons and findings from other studies underscore these infrastructural issues as significant obstacles to efficient vaccine management and the operation of VLMIS [13,15]. These challenges could severely impact the supply chain, compromising vaccine potency to the last mile and the effectiveness of its information management system [11]. The study also revealed that nearly 17% of the facilities failed to record temperatures in the cold room regularly, as recommended by the WHO [11].
A well-designed supply chain system ensures the capture of reliable and timely information through standardized logistics recording forms [5]. Among the 15 VLMIS forms intended for use in surveyed health facilities, the average utilization rate was 40.5%. The VRF showed higher availability and utilization compared to the “bin card” and “stock card,” which were almost entirely unused. This discrepancy highlights gaps in the design and implementation of VLMIS at public health facilities. Similar challenges with VLMIS have been reported in studies conducted in Northwest Ethiopia [20,21] and Uganda [22]. These issues with VLMIS forms can significantly impact data quality and, ultimately, the effectiveness of the immunization program [11,14,15].
This study evaluated the quality of VLMIS across four dimensions. Among the facilities surveyed, only 84 (84.8%) used the vaccine stock management ledger book, and merely 42 (50%) of those updated it prior to visits. The average inventory accuracy rate for the three tracer vaccines—pentavalent, BCG, and measles—was 58.3% ± 45.1%. Additional research identifies several challenges, including a shortage of recording formats, limited capacity of EPI managers, poor data visibility, and obstacles to digitalization, as significant contributors to the poor quality of inventory data [18,19,24]. The gap in vaccine stock management and the low inventory accuracy could lead to incorrect demand forecasting, procurement, and distribution, potentially affecting many lives and causing high vaccine wastage [25].
Health facilities are required to submit monthly consumption reports and demand requests to the Woreda health office or EPSS branches [26]. A 1-year review of monthly VRF reports revealed that 13 facilities (13.1%) failed to submit a report for the most recent reporting month, and 44 (44.4%) did not submit a report for the latest month. Furthermore, only 65 (65.7%) of the facilities submitted their VRF reports on time for the recent monthly report, and 41 (41.4%) did so for the latest report. This inconsistency in the timeliness of reports could lead to interruptions in vaccine supply, potentially impacting the goals of vaccination programs, as shown by multiple studies [9,11,14,15,24,27].
In low- and middle-income countries, data quality often suffers due to incomplete logistics data elements in reports [28]. This study found that VRFs were used inconsistently between urban and rural facilities. The average completeness of eight tracer medicines across seven data elements was 88.4% in facilities using the urban VRF, compared to 87.8% in those using the rural VRF. Such inconsistencies and gaps in report form completeness can adversely affect vaccine demand forecasting and supply planning, since each data element in the VRF is essential for informed decision-making [13].
The accuracy of both urban and rural VRF reports was evaluated. Facilities employing the urban VRF reported an overall accuracy of 73.4%, whereas those using the rural VRF reported an accuracy of 69.6%. Low accuracy in VRF reporting could compromise the efficiency of the supply chain and result in incorrect decision-making. Additionally, reports from the Ethiopian Ministry of Health and other studies have identified poor data quality as a major concern within the healthcare system [15,24,27].
Poor data quality in VLMIS can disrupt the supply chain and ultimately impact the lives of children. This issue may stem from the fact that all health facilities currently rely on manual VLMIS. Although implementing digital VLMIS necessitates additional infrastructure and ICT materials, it has been shown to enhance data quality and improve supply chain efficiency, as evidenced by experiences in Tanzania and other locations [29,30].
The vaccine supply chain, characterized by unpredictable demand and the need for specialized cold chain systems, must be managed by professionals with specific expertise [11]. However, this study revealed that the vaccine supply chains at health facilities were often managed by professionals lacking relevant pre-service training. EPI focal persons also identified deficiencies in regular supportive supervision and feedback from higher levels. The inconsistent use of VLMIS forms, coupled with insufficient training in vaccine management, has markedly affected the effectiveness of the immunization program. This underscores the urgent need for a standardized monitoring and evaluation framework for the VLMIS [15].
In conclusion, a robust health system necessitates a well-integrated supply chain and LMIS. Despite Ethiopia’s endeavors to enhance its health commodities supply chain, this study highlights significant deficiencies in the VLMIS at public health facilities. The identified bottlenecks encompass inconsistencies and shortages of VLMIS forms, an absence of digitalization, substandard data quality, infrastructural deficiencies, inadequate supportive supervision, and staffing challenges. These findings emphasize the critical need to standardize the VLMIS and improve data quality.
The study’s limitations are that the findings may not be generalizable to other regions due to differences in health infrastructure and local challenges specific to each context. Additionally, the lack of an electronic recording system for VLMIS could lead to errors in data abstraction from physical records, as it may be susceptible to human error.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Full text via PMC
Download Citation
