The purpose of this study was to review evaluation studies of nursing management information systems (NMISs) and their outcome measures to examine system effectiveness.
For the systematic review, a literature search of the PubMed, CINAHL, Embase, and Cochrane Library databases was conducted to retrieve original articles published between 1970 and 2014. Medical Subject Headings (MeSH) terms included informatics, medical informatics, nursing informatics, medical informatics application, and management information systems for information systems and evaluation studies and nursing evaluation research for evaluation research. Additionally, manag* and admin*, and nurs* were combined. Title, abstract, and full-text reviews were completed by two reviewers. And then, year, author, type of management system, study purpose, study design, data source, system users, study subjects, and outcomes were extracted from the selected articles. The quality and risk of bias of the studies that were finally selected were assessed with the Risk of Bias Assessment Tool for Non-randomized Studies (RoBANS) criteria.
Out of the 2,257 retrieved articles, a total of six articles were selected. These included two scheduling programs, two nursing cost-related programs, and two patient care management programs. For the outcome measurements, usefulness, time saving, satisfaction, cost, attitude, usability, data quality/completeness/accuracy, and personnel work patterns were included. User satisfaction, time saving, and usefulness mostly showed positive findings.
The study results suggest that NMISs were effective in time saving and useful in nursing care. Because there was a lack of quality in the reviewed studies, well-designed research, such as randomized controlled trials, should be conducted to more objectively evaluate the effectiveness of NMISs.
As the number of hospital information systems (HISs) have rapidly increased, systems for nursing also have increased. Since nursing care is a major operating cost within a hospital budget, nursing management is important for cost saving, and it contributes to the financial stability of hospitals [
The outcome of investment should be justified via evaluation for effectiveness in terms of various factors, such as money, time, and resources involved in the development and implementation of systems [
Although several studies evaluating the effectiveness of NMISs have been conducted, to our knowledge, there has been no summarization or synthesis of the existing evidence. Therefore, the purpose of the current study was to systemically review and synthesize the evidence on the effectiveness of NMISs used by nurses in clinical settings.
An extensive search for articles published from 1970 to May of 2014 was conducted using the PubMed, CINAHL, Embase, and Cochrane Library databases. Because all of the databases we used include terms based on Medical Subject Headings (MeSH) terms, we selected MeSH terms first, including (informatics OR nursing informatics OR medical informatics OR medical informatics application OR management information systems) for information systems, and (evaluation studies OR nursing evaluation research) for evaluation research. With those MeSH terms, we also combined other keywords, such as (manag* OR admin*) for management, and nurs* so that these wildcards could retrieve all relevant articles. Additionally, only studies written in English were used for the current study. The titles and abstracts returned by the search were read and assessed by two reviewers: one reviewer was a nursing professor who majored in nursing informatics, and the other reviewer was a doctoral student of nursing who has trained for systematic review.
An article was included if it met the following criteria: 1) included a nursing management system or a system developed as a part of nursing management; 2) was original research; and 3) included nurses as system users or study subjects. An article was excluded if it met the following criteria: 1) focused on systems used in nursing homes, the community, or long-term care facilities; 2) simply evaluated an IT device; 3) evaluated the technical aspects of a developed system; 4) evaluated systems not directly related to nursing; and 5) was a thesis, abstract, or part of conference proceedings.
From the six selected studies, we extracted type of management system, study purpose, study design, data source, system users, and study subjects. Since system users and study subjects were not necessarily the same, we extracted both types of information. Additionally, we extracted outcomes used to evaluate the effectiveness of NMISs. In the case of disagreement, differences were resolved through discussion between the two reviewers.
Study quality was independently assessed by the two reviewers using the Risk of Bias Assessment Tool for Non-randomized Studies (RoBANS 2.0) criteria [
The initial search retrieved a total of 2,257 studies: 807 from PubMed, 812 from CINAHL, 625 from Embase, and 13 from the Cochrane Library. From these, 253 duplicate articles were removed. Based on the inclusion and exclusion criteria, two members of the research team independently reviewed each article and reached a consensus regarding its exclusion. The review process for the selected articles progressed in three stages, including title review, abstract review, and full text review. We extracted 1,929 studies from the title and abstract review and 69 studies from the full text review. Finally, a total of six articles were selected for this study. The retrieval and screening process is summarized in
The selected studies included three types of NMISs. There were two scheduling programs including a perioperative system and a self-scheduling system [
With regard to study design, we found two quantitative studies [
For system users and study subjects, although the majority of users of the perioperative system were nurses, other professionals (e.g., physicians and lab technicians) were also system users [
The outcome measures of NMISs in the six studies were classified into eight categories, including usefulness, time saving, satisfaction, cost, attitude, usability, data quality/completeness/accuracy, and personnel work patterns. Most studies used multiple outcome measures, ranging from 2 to 7, with an average of 5.0 per study (
All six studies evaluated 'usefulness', and most studies addressed the positive results. For example, nursing financial management systems helped make the nursing staffs' work processes less complicated and improved productivity [
'Time saving' and 'satisfaction' outcomes were the second most evaluated measurements, and they were included in five studies [
In the 'satisfaction' category, nurse managers were satisfied with the nursing financial management system, which ensured that delays in information reporting did not occur [
'Cost' was evaluated as an outcome measurement in four studies. The nursing financial management system eliminated salaries related to re-working, which led to 122% of return on investment [
'Attitude' was evaluated as an outcome measurement in three studies. In the study of the nursing resource management information system, the mean score of implementation on attitude (i.e., job performance) was 13.3, with a range of -26 to +26 [
'Usability' was also evaluated as an outcome measurement in three studies. In the nursing resource management information system study, the mean ease of use score was 16.3, with a range of -18 to +18 [
Three of the selected studies measured the outcome of 'data quality/completeness/accuracy'. With regard to the nursing resource management information system, the mean score of information accuracy was 8.3, with a range of 2-10 [
Two of the selected studies measured the outcome of 'personnel work patterns'. The perioperative system helped to track continuing staff education and basic life support renewal dates [
The quality of the included studies is summarized in
We conducted a systematic review of the studies that have evaluated various NMISs in terms of methods and outcome measures. We attempted to show not only the methods and outcome measures of the evaluation studies, but also the positive or negative aspects of outcomes from the six articles that met the inclusion criteria from our literature search.
Evaluations assessed two scheduling programs, two nursing cost-related programs, and two patient care management programs. Half of the studies assessed utilized quantitative and qualitative mixed-method designs (n = 3), and only one study adopted a test group and control group comparison approach [
Among data collection methods, questionnaires and chart/EMR reviews were frequently used, which is similar to findings from other studies [
In addition, most studies had multiple evaluation outcomes, with an average number of outcomes per study of 5.3. This is consistent with Ammenwerth and de Keizer [
In the current study, the quality of selected studies was unclear for most of the risk of bias criteria in the RoBANS appraisal tool. Friedman and Abbas [
In addition, we had some difficulty categorizing the NMISs. This may be an inherent issue with review studies that attempt to categorize systems into certain types because systems may have multiple aspects or dominant/non-dominant functions. Consequently, we considered the dominant aspect of information systems and categorized them accordingly to achieve the best fit; however, some readers might disagree with our categorizations. In addition, there were so few studies that developed and evaluated the information systems focused on nursing management that the quality of the selected studies was not certain.
Since NMISs were executed within complex and dynamic hospital environments, an issue that may arise is the interpretation of results based on the application of certain viewpoints in various studies. For example, in terms of evaluating time saving, the substantial number of data elements required by information systems was negatively evaluated [
According to our review of the studies related to the evaluation of NMISs in clinical settings, a study with a dynamic and sufficient design including long-term follow-up (i.e., a longitudinal study) and patient outcomes has not yet been conducted. As a result, a plan for evaluation should be integrated at the beginning of the information development process [
This study was supported by a part of research fund from College of Nursing, the Catholic University of Korea.
No potential conflict of interest relevant to this article was reported.
Literature searching flow. NMIS: nursing management information system.
Summary of NIMS evaluation studies
aMixed: quantitative and qualitative mixed method study design.
Outcomes and results of effectiveness of NMISs
NMIS: nursing management information system, CNDMS: Computer Nursing Dependency Management System, NPR: nurse patient ratio.
Quality of studies
○: low risk of bias, Ⅹ: high risk of bias, △: unclear.