Evaluation of User Experience of New Defense Medical Information System

Article information

Healthc Inform Res. 2019;25(2):73-81
Publication date (electronic) : 2019 April 30
doi : https://doi.org/10.4258/hir.2019.25.2.73
1College of Nursing, Seoul National University, Seoul, Korea.
2Research Institute of Nursing Science, Seoul National University, Seoul, Korea.
Corresponding Author: Jeongeun Kim. College of Nursing, Seoul National University, 103 Daehak-ro, Jongno-gu, Seoul 03080, Korea. Tel: +82-2-740-8483, kim0424@snu.ac.kr
Received 2018 December 24; Revised 2019 February 01; Accepted 2019 February 07.

Abstract

Objectives

This study aimed to investigate the user experience (UX) of the New Defense Medical Information System (N-DEMIS), which was introduced in 2012 as part of an effort to improve the old system of armed forces hospitals and ultimately bring their standards up to those of civilian hospitals.

Methods

In this study, the dependent variable was the UX of N-DEMIS and was composed of usability, affect, and user value. The questionnaire comprised 41 questions: nine on general characteristics, 20 on usability, four on affect, and eight on user value. The data collection period was from April 15 to April 30, 2018. Overall, 85 responses were received; of these, three insincere responses were excluded, and the remaining 82 responses were used in the analysis.

Results

The overall value of Cronbach's alpha was 0.917, indicating an overall high-reliability. There was a significant difference between user value and usability, but there was no significant differences between the other pairs. We observed a significant effect on UX for length of time working in an armed forces hospital and employment type.

Conclusions

The results of our survey showed an even distribution of scores across the three elements of UX, showing that no particular aspect of N-DEMIS is superior to the others in terms of user satisfaction. However, the overall UX score of around 60% indicates the need for future improvements. Rather than focusing improvements on a specific area, improvements should be spread across usability, affect, and user value.

I. Introduction

In civilian medical institutions, medical systems are rapidly developed in response to dynamic social needs and seek to reduce gaps, and armed forces hospitals have been accelerating the development of efficient information and communication technology management. One aspect of this effort is the Defense Medical Information System (DEMIS), which has been used by armed forces hospitals since its introduction in 2000. In 2012, the original DEMIS was modified to produce a next-generation, web-based integrated medical information system to enable standardized work across all armed forces. The new system called the New Defense Medical Information System (N-DEMIS) is currently in operation [1]. DEMIS is a hospital information system (HIS) used by the South Korea military, and N-DEMIS is the name of the new version of this program. N-DEMIS is a closed system that does not interoperate with other hospital systems in Korea with mandatory military service. It is independently operated and limits the application of existing Electronic Medical Record (EMR) and HIS results. Therefore, research on N-DEMIS is needed.

User experience (UX) encompasses all direct and indirect experiences of the user while interacting with a product or service [23]. Although there are still conflicting opinions about specific elements that comprise UX, most researchers agree that it is an overarching concept that includes usability and affect [34]. Previously, in the field of human-computer interaction (HCI), the methodology to assess products and services involved the quantitative investigation of problem areas through usability testing, which focuses on issues such as physical comfort or ease of operation. However, usability only considers the point of contact between human and system, such as whether a task is completed and how long it takes. Because understanding user experience requires a comprehensive analysis of the user's entire experience while using a system, usability alone will not meet the requirements for assessing current tasks within the future focus of HCI on UX [456].

Although there has been much debate regarding the differences between usability and UX, the definitions provided by the International Organization for Standardization (ISO) offer some of the most useful insights. The ISO defines usability as “the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments”, whereas UX is defined as a “person's perceptions and responses resulting from the use and/or anticipated use of a product, system, or service”. The Nielsen Norman Group, a large UX consulting firm, states that UX “includes all aspects of the end-user's interaction with a company, service, or product” [67].

Although the term UX includes various concepts, it has not been clearly defined. The assimilation of existing UX research has generated a definition based on three elements: usability, affect, and user value [78910]. These elements were further divided into seven sub-elements of usability, six subelements of affect, and five sub-elements of user value [11].

The Korean peninsula is technically in a state of truce. All medical personnel of the Republic of Korea are to be dispatched to field hospitals and other medical institutions if a war breaks out. In addition, every male over 18 years of age is obliged to serve in the military for 2 years. Most medical students serve as military officers after acquiring a medical license as a doctor or a specialist [12]. The use of hospital information systems has a great impact on the performance of medical personnel [13]. Therefore, the quality of the information system is crucial [1415]. Well-designed medical information systems would contribute to better patient outcomes, whereas poorly designed systems would have a negative effect [15161718], burdening medical staff with unnecessary extra stress [19]. In this particular situation in Korea, it is very important to evaluate the N-DEMIS that will be used by many Korean medical personnel in emergencies such as war. To date, related studies have evaluated only usability, whereas this study measured user experience [2021].

To provide basic data for effective use of N-DEMIS in the future, this study analyzed and evaluated the UX of the current N-DEMIS through comparison of components of UX, comparison of UX by subject characteristics, and analyzed the variables that affect UX.

II. Methods

1. User Experience and Its Elements

In this study, the dependent variable was the UX of N-DEMIS and was composed of usability, affect, and user value. The independent variables were the subject characteristics of age, highest educational attainment, title, rank, length of career, length of clinical experience, length of time using N-DEMIS, length of time using systems other than N-DEMIS, and employment type.

In a previous study, usability was defined in terms of seven sub-elements, namely, simplicity, directness, efficiency, informativeness, flexibility, learnability, and user support. These sub-elements are related to the basic usability of the program and have been specified in previous studies [12]. A previous study on affect suggested dividing affect simply into two dimensions and described affective quality not as quality perceived by a person, but rather as quality inherent in an object [22]. In the present study, we applied concepts adapted from a previous study [10]. Of the six sub-elements suggested in the previous study, we excluded delicacy and texture because these were considered unsuitable for our aim of assessing a medical information system and measured only the four remaining sub-elements of simplicity, luxuriousness, color, and attractiveness. Using only the conventional elements of usability and affect, it is difficult to accurately measure the personal and subjective features of UX. As a result, user value has begun to emerge as another element of UX [23]. User value can be defined as the subjective value granted to a product by the user. This concept is also related to how significant or important the user perceives the product to be in his or her life. Adapted from the sub-elements of a previous research, the user value in our study was composed of the four sub-elements of self-satisfaction, pleasure, sociability, and customer need [10].

2. Data Collection

The subjects were individuals working at an armed forces hospital and using N-DEMIS. They read an online description of the study and consented to participate in the study via an online form. Subjects should have been using the system for at least 6 months at the time of questionnaire completion.

The survey was advertised at all 13 military hospitals of South Korea. In the advertisement, it was explained that the subjects' anonymity would be guaranteed and that they could withdraw from the study at any time they wished. All subjects gave their consent voluntarily before completing the online questionnaire. Data collection was carried out from April 15 to April 30, 2018. Overall, 85 responses were received, and of these, three inadequate responses were excluded. The responses of the remaining 82 subjects were used in the analysis.

3. Study Questionnaire for User Experience Measurement

The questionnaire comprised 41 questions: nine on subject characteristics, 20 on usability, four on affect, and eight on user value. For subject characteristics, we measured the nine variables of age, highest educational attainment, title, rank, length of time working at an armed forces hospital, length of clinical experience, length of time using N-DEMIS, length of time using similar systems other than N-DEMIS, and employment type (e.g., compulsory military service). Except subject characteristics, all other questions were scored on a 5-point scale. UX as the dependent variable was the sum of the three components of usability, affect, and user value; therefore, it could have a maximum score of 15 points for each subject, and each of the three components was assessed as the mean of response values from the questions.

The questionnaire was constructed based on a usability study on DEMIS, but it was developed further to reflect the three UX elements and their sub-elements. The questionnaire was constructed by matching the definitions of the three UX components and their sub-elements against content from the instrument used to evaluate the usability of the previous version of the system being studied [24]. The content validity index (CVI) evaluation of the draft questionnaire was carried out by relevant experts. Three nurses and two doctors participated in the evaluation. All of the evaluators had higher qualifications than a master's degree in medicine or nursing. They also had experience using N-DEMIS or a number of other hospital information systems for more than 3 years. We ensured that only validated questions (≥0.8) were used in the study.

4. Methods of Analysis

For data processing, IBM SPSS Statistics version 25 for Windows (IBM Corp., Armonk, NY, USA) was used for descriptive statistics and hypothesis testing. To evaluate the reliability of the questionnaire, we used Cronbach's alpha to analyze the internal consistency of each item in the UX. Descriptive statistics and frequency analysis were used to investigate the distribution of subject characteristics, and t-tests were applied to investigate differences between usability, affect, and user value. In addition, to compare UX for the subgroups based on each subject characteristic, we performed t-tests, ANOVA, and multiple comparison tests. In the case of continuous variables of subject characteristics, the subjects were divided into subgroups considering the distribution of the variables. UX were tested to compare each subgroup which were divided into 3 based on subject characteristics. The three components were tested by t-test and ANOVA.

To test how each user characteristic affects the UX of N-DEMIS, a multiple linear regression analysis was conducted. Dummy variables were used for categorical variables, and all variables of user characteristics were tested as independent variables.

III. Results

1. Questionnaire Reliability

The Cronbach's alpha values for usability, affect, and user value were 0.887, 0.743, and 0.728, respectively. In this study, all elements showed a Cronbach's alpha of 0.7 or higher, and the overall Cronbach's alpha was 0.917, indicating an overall high-reliability.

2. Subject Distribution

Among the 82 responding subjects, the majority were nursing officers (76.8%), with the remainder being chief nursing officers (11.0%), medical officers (3.7%), and others (8.5%). Considering rank, first lieutenants, second lieutenants, and captains were 62.2%, 1.2%, and 31.7%, respectively, of the total. The rest of the subjects were civilian personnel (4.9%). The age (mean ± standard deviation) was 27.23 ± 3.77 years, the mean length of time working at an armed forces hospital was 2.87 ± 1.696 years, the mean length of clinical experience was 3.10 ± 1.796 years, the mean length of time using N-DEMIS was 2.80 ± 1.543 years, and the mean length of time using systems other than N-DEMIS was 1.85 ± 1.895 years. Almost of all the subjects had undergraduate degrees as their highest educational attainment (93.9%). Others had 3-year undergraduate program college degrees (2.4%) or graduate or higher degrees (3.7%) for their highest educational attainment. The major employment type was compulsory military service (69.5%), and the rest of the subjects were in extended or long-term service (25.6%) or other types, representing contract employees, such as civilian personnel (4.9%).

3. Differences in Each Component of User Experience

Table 1 shows the means and standard deviations for each element of UX. Among the elements of UX, the mean scores for usability and affect were similar, at 3.11 and 3.10, respectively, while the mean score for user value was the lowest, at 3.00. Examination of the standard deviations shows that the distribution of scores across users was relatively even.

Table 1

Questions and results for measurement of user experience and each component

We performed a t-test to analyze the differences between the elements as seen in Figure 1. The p-values for usability-affect, user value-usability, and affect-user value were 0.905, 0.046, and 0.098, respectively, indicating a significant difference between user value and usability, and no significant differences between the other pairs.

Figure 1

Distribution of scores in each element of user experience.

4. Analysis of User Experience by Subject Characteristics

We also tested for differences in UX depending on the subgroups for each of the measured subject characteristics, and the results are shown in Table 2. Continuous variables were divided into subgroups based on distribution, as shown in Table 2. We observed significant differences in UX in relation to length of time working in an armed forces hospital (p = 0.050) and employment type (p = 0.010). Length of time working in an armed forces hospital did not show a significant relationship in the correlation analysis (p =0.549), so there was no specific trend, but the UX scores tended to be lower for subjects who had between 2 and 5 years of experience. For employment type, the highest UX scores were shown by the compulsory military service group, while the lowest UX scores were shown by the other group. From the multiple comparison test, the compulsory military service group had a significant difference from the extended or long-term service group (p = 0.048); however, there was no significant difference from the other group (p = 0.104).

Table 2

Differences in user experience according to subject characteristics

When each component was assessed in relation to subject characteristics, all three components of UX (usability, affect, and user value) differed significantly according to the employment type, and compulsory military service showed the highest scores for all three components, which is aligned with the UX results.

A multiple linear regression analysis with backward elimination was conducted, and the variables used are shown in Table 3. The ANOVA for regression showed that the model was significant at the 0.003 probability level, and R2 was equal to 0.188. Among the highest educational attainments, only the 3-year undergraduate program college degree showed significance in the regression analysis. In addition to educational attainments, the length of time working in an armed forces hospital and employment types were selected as determinant factors for UX of N-DEMIS.

Table 3

Results of estimating variables for user experience by multiple linear regression

IV. Discussion

The subjects were individuals working in a hospital-level armed forces medical institution and using N-DEMIS for their work. To analyze UX, we broke it down into the elements of usability, affect, and user value. The results of our survey showed an even distribution of scores across the three elements of UX, showing that no particular aspect of N-DEMIS is superior to the others in terms of user satisfaction. Since there were no major differences between elements, no particular aspect of the N-DEMIS UX can be considered significantly more important than the others. Even though the difference between usability and user value was statistically significant, it was only a small difference considering the standard deviations of measurements.

We analyzed the factors that affect UX or each component of UX; however, there were no differences according to the subject characteristics of age or education or clinical characteristics of clinical experience and experience using EMR. Also, clinical experience or experience using EMR did not show any differences. There have been previous studies in the usability of HIS that found differences in relation to the subjects' characteristics. However, direct comparisons between the previous studies and this study would be problematic because the measured values are not comparable [24]. It should be noted that the dataset may not have been big enough to show differences in UX in relation to the mentioned factors. Conversely, factors specific to armed forces hospitals, such as length of time working in an armed forces hospital and employment type, were associated with differences in UX. These results are consistent with the results of similar studies [2425]. In particular, employment type was a significant variable for all sub-elements of usability, affect, and user value. These results are consistent with those of similar studies [26], which claimed that the length of time using a system can affect the usability. Although direct comparison should be made with caution, it can be said that better job satisfaction of senior staff generally resulted in better system satisfaction.

Even the multiple regression analysis would partially explain the influencing variables on UX; with relatively low R2 value, the results were found to be aligned with the results of each characteristic's analysis. Working time in an armed forces hospital and employment type were also analyzed as significant variables as in the comparison analysis mentioned above. Although the highest educational attainment could be identified as a significant variable, only two subjects had 3-year undergraduate program college degrees. The ANOVA result for the influence of the highest educational attainment on UX did not show significance (p = 0.646); therefore, education level should not be considered an influencing factor for UX of N-DEMIS.

Due to the nature of armed forces hospitals, patient groups and treatment types are relatively similar [27]. This means that users showed little difference in the usage of the system, which could be a reason for the homogeneity of UX between users [28]. However, the overall UX score of around 60% indicates the need for future improvements.

There are many ways to evaluate UX, but this study was based on only the survey method using a questionnaire. The majority of the subjects were junior nursing officers and did not reflect the variety of actual users who interact with the system [29]. Therefore, it is important to note that the results should not be generalized. In addition, it should be noted that the questionnaire used in the survey had not been used before. Although the reliability and validity of the questionnaire were verified, they had not been tested in other studies, and the construct validity was not verified either. In future research, it is recommended to study the UX of various subjects using a variety of verified tools. Rather than focusing improvements on a specific area, improvements should be spread across usability, affect, and user value.

In this study, we did not analyze specific problems in N-DEMIS or perform a cost-benefit analysis. Since this was a survey study, we were not able to improve the UX of the system; therefore, there should be some consideration for future system improvements. In addition to the existing user base for N-DEMIS, in the event of war or disaster, the system needs to be rapidly learned and adopted by new wartime medical personnel. Therefore, user affinity and UX need to be even higher than they are for systems used in the civilian sector. For these reasons, it is important to continually evaluate UX, analyze factors affecting UX, and prepare measures to improve UX. This will require policies from related institutions to improve the system and provide standardized management of work using the system. In addition, it is important to check whether the service has improved for troops who are the direct and indirect beneficiaries of the system and whether there has been any change in the quality of medical care and nursing.

Notes

Conflict of Interest: No potential conflict of interest relevant to this article was reported.

References

1. Rhee C, Burkom H, Yoon CG, Stewart M, Elbert Y, Katz A, et al. Syndromic surveillance system for Korea-US joint biosurveillance portal: design and lessons learned. Health Secur 2016;14(3):152–160. 27314655.
2. Desmet P, Hekkert P. Framework of product experience. Int J Des 2007;1(1):57–66.
3. Law EL, Van Schaik P. Modelling user experience: an agenda for research and practice. Interact Comput 2010;22(5):313–322.
4. Lee K, Lee I, Jun S, Yang S, Choi GW, Kim J, et al. How to measure the user experience? In : Proceedings of HCI Society of Korea; 2008 Feb 13–15; Pyungchang, Korea. p. 851–856.
5. Lee SM. On the UX of social commerce shopping application. Illus Forum 2014;39:5–14.
6. Norman D, Nielsen J. The definition of user experience (UX) [Internet] San Diego (CA): Nielsen Norman Group; c2019. cited at 2019 Apr 19. Available from: https://www.nngroup.com/articles/definition-userexperience/.
7. Kim HK, Han SH, Park J, Park W, Park YS, Cho Y, et al. The definition of user experience through a literature survey In : Proceedings of the 2009 Fall Conference of the Korean Institute of Industrial Engineers; 2009 Oct 9; Yongin, Korea. p. 305–309.
8. Hassenzahl M, Tractinsky N. User experience-a research agenda. Behav Inf Technol 2006;25(2):91–97.
9. Thuring M, Mahlke S. Usability, aesthetics and emotions in human–technology interaction. Int J Psychol 2007;42(4):253–264.
10. Park J, Han SH, Kim HK, Cho Y, Park W. Developing elements of user experience for mobile phones and services: survey, interview, and observation approaches. Hum Factors Ergon Manuf Serv Ind 2013;23(4):279–293.
11. Hong SW. A methodology for modelling and analyzing user's affective satisfaction toward consumer electronic products [dissertation] Pohang, Korea: Pohang University of Science and Technology; 2005.
12. Kim MH, Park H, Park EC, Park K. Attitude and knowledge of physicians about cancer pain management: young doctors of South Korea in their early career. Jpn J Clin Oncol 2011;41(6):783–791. 21502282.
13. Staggers N, Jennings BM, Lasome CE. A usability assessment of AHLTA in ambulatory clinics at a military medical center. Mil Med 2010;175(7):518–524. 20684457.
14. Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011;8(1)e1000387. 21267058.
15. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144(10):742–752. 16702590.
16. Blumenthal D, Glaser JP. Information technology comes to medicine. N Engl J Med 2007;356(24):2527–2534. 17568035.
17. Ayatollahi H, Bath PA, Goodacre S. Factors influencing the use of IT in the emergency department: a qualitative study. Health Informatics J 2010;16(3):189–200. 20889849.
18. Stevenson JE, Nilsson GC, Petersson GI, Johansson PE. Nurses' experience of using electronic patient records in everyday practice in acute/inpatient ward settings: a literature review. Health Informatics J 2010;16(1):63–72. 20413414.
19. Heponiemi T, Hypponen H, Vehko T, Kujala S, Aalto AM, Vanska J, et al. Finnish physicians' stress related to information systems keeps increasing: a longitudinal three-wave survey study. BMC Med Inform Decis Mak 2017;17(1):147. 29041971.
20. Burkhard R, Fruhling A, Djamasbi S. Introduction to user experience (UX) in information systems (IS) for health and wellness minitrack In : Proceedings of the 50th Hawaii International Conference on System Sciences; 2017 Jan 4–7; Hilton Waikoloa Village, HI.
21. Staggers N, Xiao Y, Chapman L. Debunking health IT usability myths. Appl Clin Inform 2013;4(2):241–250. 23874361.
22. Russell JA. Core affect and the psychological construction of emotion. Psychol Rev 2003;110(1):145–172. 12529060.
23. Hassenzahl M. The thing and I: understanding the relationship between user and product. In : Blythe MA, Overbeeke K, Monk AF, et al, eds. Funology Dordrecht: Springer; 2003. p. 31–42.
24. Lee SK. A study on the user satisfaction of the DEMIS [dissertation] Seoul, Korea: Yonsei University; 2001.
25. Khalifa M, Alswailem O. Hospital information systems (HIS) acceptance and satisfaction: a case study of a tertiary care hospital. Procedia Comput Sci 2015;63:198–204.
26. Park DE. Nursing work environment, job satisfaction, and turnover intention of military nursing officer in armed forces hospital [dissertation] Seoul, Korea: Seoul National University; 2014.
27. Jung HE. The determinants of inpatients' satisfaction at military hospitals in Korea [dissertation] Seoul, Korea: Yonsei University; 2005.
28. Yoo MR, Jung YM, Kim SY, Kim MJ. Re-employment experiences of retired nurse officers. J Mil Nurs Res 2012;30(2):16–30.
29. Yoo MR, Choi YJ, Yoo JA. The study on competency of junior nursing officers. J Mil Nurs Res 2010;28(2):29–44.

Article information Continued

Figure 1

Distribution of scores in each element of user experience.

Table 1

Questions and results for measurement of user experience and each component

Table 1

Values are presented as mean ± standard deviation.

N-DEMIS: New Defense Medical Information System.

Table 2

Differences in user experience according to subject characteristics

Table 2

Values are presented as mean ± standard deviation.

N-DEMIS: New Defense Medical Information System.

Table 3

Results of estimating variables for user experience by multiple linear regression

Table 3