An extensive search for articles published from 1970 to May of 2014 was conducted using the PubMed, CINAHL, Embase, and Cochrane Library databases. Because all of the databases we used include terms based on Medical Subject Headings (MeSH) terms, we selected MeSH terms first, including (informatics OR nursing informatics OR medical informatics OR medical informatics application OR management information systems) for information systems, and (evaluation studies OR nursing evaluation research) for evaluation research. With those MeSH terms, we also combined other keywords, such as (manag^* OR admin^*) for management, and nurs^* so that these wildcards could retrieve all relevant articles. Additionally, only studies written in English were used for the current study. The titles and abstracts returned by the search were read and assessed by two reviewers: one reviewer was a nursing professor who majored in nursing informatics, and the other reviewer was a doctoral student of nursing who has trained for systematic review.

An article was included if it met the following criteria: 1) included a nursing management system or a system developed as a part of nursing management; 2) was original research; and 3) included nurses as system users or study subjects. An article was excluded if it met the following criteria: 1) focused on systems used in nursing homes, the community, or long-term care facilities; 2) simply evaluated an IT device; 3) evaluated the technical aspects of a developed system; 4) evaluated systems not directly related to nursing; and 5) was a thesis, abstract, or part of conference proceedings.

From the six selected studies, we extracted type of management system, study purpose, study design, data source, system users, and study subjects. Since system users and study subjects were not necessarily the same, we extracted both types of information. Additionally, we extracted outcomes used to evaluate the effectiveness of NMISs. In the case of disagreement, differences were resolved through discussion between the two reviewers.

Study quality was independently assessed by the two reviewers using the Risk of Bias Assessment Tool for Non-randomized Studies (RoBANS 2.0) criteria [13]. The RoBANS criteria cover seven specific areas, including comparability of participants, selection of participants, confounding variables, intervention measurement, blinding of outcome assessment, outcome evaluation, incomplete outcome data, and selective outcome reporting [13]. Each criterion was evaluated as 'low risk of bias', 'high risk of bias', or 'unclear'. If the study did not mention a certain criterion, we evaluated it as 'unclear'. In instances of disagreement, each case was discussed with all authors.

The initial search retrieved a total of 2,257 studies: 807 from PubMed, 812 from CINAHL, 625 from Embase, and 13 from the Cochrane Library. From these, 253 duplicate articles were removed. Based on the inclusion and exclusion criteria, two members of the research team independently reviewed each article and reached a consensus regarding its exclusion. The review process for the selected articles progressed in three stages, including title review, abstract review, and full text review. We extracted 1,929 studies from the title and abstract review and 69 studies from the full text review. Finally, a total of six articles were selected for this study. The retrieval and screening process is summarized in Figure 1.

The selected studies included three types of NMISs. There were two scheduling programs including a perioperative system and a self-scheduling system [11,12] as well as two nursing cost-related programs, including a nursing resource management system and a nursing financial management system [6,9]. Moreover, there were two patient care management programs, including a data warehouse-based NMIS and a computerized nurse dependency management system [4,10] (Table 1).

With regard to study design, we found two quantitative studies [9,10], three mixed studies with both quantitative and qualitative approaches [4,6,11], and a descriptive study [12]. For the data source, questionnaires were the most preferred quantitative method, and they were used in four studies [4,6,9,11]. Qualitative approaches, including semi-structured interview and focus group interview were used in three studies [4,6,11]. Additionally, hospital data, such as fiscal and human resource data [9] as well as electronic nursing workload management reports [10], were utilized to evaluate system effectiveness.

For system users and study subjects, although the majority of users of the perioperative system were nurses, other professionals (e.g., physicians and lab technicians) were also system users [12]. The study subjects were operating room nurses, unit nurse managers or directors, unit head nurses, and experts in clinical nursing informatics (Table 1).

The outcome measures of NMISs in the six studies were classified into eight categories, including usefulness, time saving, satisfaction, cost, attitude, usability, data quality/completeness/accuracy, and personnel work patterns. Most studies used multiple outcome measures, ranging from 2 to 7, with an average of 5.0 per study (Table 2).

All six studies evaluated 'usefulness', and most studies addressed the positive results. For example, nursing financial management systems helped make the nursing staffs' work processes less complicated and improved productivity [9]. In addition, the perioperative system decreased cancellations and equipment conflicts in operating rooms and improved overall documentation [12]. In the nursing resource management information system evaluation study, nurses' perceived usefulness mean score was 26.7, with a range of -33.0 to +33.0, which was a high score [6]. Furthermore, the data warehouse-based nursing management system improved care to meet actual care needs (40%), and appropriate care was delivered at the right time according to predefined clinical processes (20%) [4]. By using the computerized nurse dependency management system, nurse managers allowed for continuous patient dependency data to be used to allocate and predict staff allocation based on nursing care requirements [10]. However, one negative result was described in terms of the self-scheduling system study. Specifically, in this study, nurses could not predict what events would happen 2-3 months ahead of time, and self-scheduling was difficult when the schedule had to be changed [11].

'Time saving' and 'satisfaction' outcomes were the second most evaluated measurements, and they were included in five studies [4,6,9,11,12]. Furthermore, most of the results demonstrated positive effects. In the time saving category, the nursing financial management system evaluation study showed that use of the system led to 6% of staff's time spent on report generation compared to 52% with the manual system. Moreover, the time required for data collection, organization, and manipulation decreased by 88%, demonstrating a savings of 2,410 hours annually [9]. In addition, the perioperative system decreased the number of hours nurses spent ordering supplies and reduced unnecessary inventory [12]. The nursing resource management information system also reduced expenditures for overtime and extra hours compared to the control group [6]. Users of the self-scheduling system perceived that they had more time to spend with their families and felt they provided better patient care as a result [11]. On the other hand, the 'time saving' outcome was associated with some negative aspects of the NMISs. For instance, the self-scheduling system created too much work for the nurse managers to organize scheduling [11]. In addition, the data warehouse-based NMIS included too much data content with several dimensions that made data exploration obscure, and it was found to be confusing and time consuming in everyday use (27%) [4].

In the 'satisfaction' category, nurse managers were satisfied with the nursing financial management system, which ensured that delays in information reporting did not occur [9]. Additionally, the perioperative system allowed for reporting of caseload, average surgical time, cost per case, room usage or turnover time, numbers of cases, average case costs, case cancellation reasons, and the number of cancellations [12]. With regard to the nursing resource management information system, nurses' mean score for satisfaction was 54.7, with a range of 12-60 [6]. In the study of the self-scheduling system, nurse users were able to control their schedules and felt more freedom in their personal lives (70%), although some competition occurred when selecting preferred shifts [11]. The data warehouse-based NMIS does not include important quality aspects of a patient's care (40%) or information describing personnel competency and educational needs (30%) [4].

'Cost' was evaluated as an outcome measurement in four studies. The nursing financial management system eliminated salaries related to re-working, which led to 122% of return on investment [9]. In addition, the perioperative system decreased lost charges [12], while the nursing resource management information system improved the budget balance [6]. Furthermore, the computerized nurse dependency management system under-predicted the decreased average number of hours per ward and per shift compared to the manual system. Moreover, the system allowed for staff allocation to meet patients' varying requirements for nursing hours and skill mix [10].

'Attitude' was evaluated as an outcome measurement in three studies. In the study of the nursing resource management information system, the mean score of implementation on attitude (i.e., job performance) was 13.3, with a range of -26 to +26 [6]. However, the self-scheduling system gradually decreased control and flexibility [11]. The data warehouse-based NMIS showed positive aspects in terms of the systematic production of information from available nursing databases [4].

'Usability' was also evaluated as an outcome measurement in three studies. In the nursing resource management information system study, the mean ease of use score was 16.3, with a range of -18 to +18 [6]. The data warehouse-based NMIS was useable (22%) and multi-professional use was available to ensure total quality of patient care (40%) [4]. However, the perioperative system study described how the personnel module only allowed users to view 10 personnel and 2 weeks of the schedule on each screen [12].

Three of the selected studies measured the outcome of 'data quality/completeness/accuracy'. With regard to the nursing resource management information system, the mean score of information accuracy was 8.3, with a range of 2-10 [6]. In the data warehouse-based NMIS study, risk of misleading conclusions was 27% if users lacked competencies in either nursing management or statistical decision-making, and 40% of participants demanded that data from other HIS-subsystems (e.g., hospital infection data) be added [4]. The computerized nurse dependency management system provided a detailed measure of the complexity of patient needs and their dependency on nurses, while also electronically recording actual care. Thus, it predicts the care required for individual patient needs and their dependency on nurses [10].

Two of the selected studies measured the outcome of 'personnel work patterns'. The perioperative system helped to track continuing staff education and basic life support renewal dates [12]. In the interviews of the self-scheduling system users, participants described how the system provided a feeling of control over their own lives and allowed them to schedule work based on their personal needs without filling out multiple time request forms [11].

The quality of the included studies is summarized in Table 3. In appraising the risk of bias as a result of inappropriate comparability and participant selection, only one study [6] was evaluated as 'low risk of bias' on comparability as it compared 4 test units and 6 control units by simultaneous parallel measurements. We evaluated studies that did not specifically describe participants as 'unclear' [4,9,10,11,12] and one study that used convenient sampling as 'high risk of bias' on participant selection [6]. All studies were also 'unclear' about blinding outcome assessment and incomplete outcome data. On the other hand, all studies did not consider confounding variables that were evaluated as 'high risk of bias' on confounding variables. In terms of the intervention measurement, three studies were rated as having a 'low risk of bias' [9,10,12] and others were appraised as having a 'high risk of bias' because of using non-standardized measurements, such as interviews [4,6,11]. Most of the criteria were 'unclear' or 'high risk of bias' on two studies [4,11].