In this study, the dependent variable was the UX of N-DEMIS and was composed of usability, affect, and user value. The independent variables were the subject characteristics of age, highest educational attainment, title, rank, length of career, length of clinical experience, length of time using N-DEMIS, length of time using systems other than N-DEMIS, and employment type.

In a previous study, usability was defined in terms of seven sub-elements, namely, simplicity, directness, efficiency, informativeness, flexibility, learnability, and user support. These sub-elements are related to the basic usability of the program and have been specified in previous studies [12]. A previous study on affect suggested dividing affect simply into two dimensions and described affective quality not as quality perceived by a person, but rather as quality inherent in an object [22]. In the present study, we applied concepts adapted from a previous study [10]. Of the six sub-elements suggested in the previous study, we excluded delicacy and texture because these were considered unsuitable for our aim of assessing a medical information system and measured only the four remaining sub-elements of simplicity, luxuriousness, color, and attractiveness. Using only the conventional elements of usability and affect, it is difficult to accurately measure the personal and subjective features of UX. As a result, user value has begun to emerge as another element of UX [23]. User value can be defined as the subjective value granted to a product by the user. This concept is also related to how significant or important the user perceives the product to be in his or her life. Adapted from the sub-elements of a previous research, the user value in our study was composed of the four sub-elements of self-satisfaction, pleasure, sociability, and customer need [10].

The subjects were individuals working at an armed forces hospital and using N-DEMIS. They read an online description of the study and consented to participate in the study via an online form. Subjects should have been using the system for at least 6 months at the time of questionnaire completion.

The survey was advertised at all 13 military hospitals of South Korea. In the advertisement, it was explained that the subjects' anonymity would be guaranteed and that they could withdraw from the study at any time they wished. All subjects gave their consent voluntarily before completing the online questionnaire. Data collection was carried out from April 15 to April 30, 2018. Overall, 85 responses were received, and of these, three inadequate responses were excluded. The responses of the remaining 82 subjects were used in the analysis.

The questionnaire comprised 41 questions: nine on subject characteristics, 20 on usability, four on affect, and eight on user value. For subject characteristics, we measured the nine variables of age, highest educational attainment, title, rank, length of time working at an armed forces hospital, length of clinical experience, length of time using N-DEMIS, length of time using similar systems other than N-DEMIS, and employment type (e.g., compulsory military service). Except subject characteristics, all other questions were scored on a 5-point scale. UX as the dependent variable was the sum of the three components of usability, affect, and user value; therefore, it could have a maximum score of 15 points for each subject, and each of the three components was assessed as the mean of response values from the questions.

The questionnaire was constructed based on a usability study on DEMIS, but it was developed further to reflect the three UX elements and their sub-elements. The questionnaire was constructed by matching the definitions of the three UX components and their sub-elements against content from the instrument used to evaluate the usability of the previous version of the system being studied [24]. The content validity index (CVI) evaluation of the draft questionnaire was carried out by relevant experts. Three nurses and two doctors participated in the evaluation. All of the evaluators had higher qualifications than a master's degree in medicine or nursing. They also had experience using N-DEMIS or a number of other hospital information systems for more than 3 years. We ensured that only validated questions (≥0.8) were used in the study.

For data processing, IBM SPSS Statistics version 25 for Windows (IBM Corp., Armonk, NY, USA) was used for descriptive statistics and hypothesis testing. To evaluate the reliability of the questionnaire, we used Cronbach's alpha to analyze the internal consistency of each item in the UX. Descriptive statistics and frequency analysis were used to investigate the distribution of subject characteristics, and t-tests were applied to investigate differences between usability, affect, and user value. In addition, to compare UX for the subgroups based on each subject characteristic, we performed t-tests, ANOVA, and multiple comparison tests. In the case of continuous variables of subject characteristics, the subjects were divided into subgroups considering the distribution of the variables. UX were tested to compare each subgroup which were divided into 3 based on subject characteristics. The three components were tested by t-test and ANOVA.

To test how each user characteristic affects the UX of N-DEMIS, a multiple linear regression analysis was conducted. Dummy variables were used for categorical variables, and all variables of user characteristics were tested as independent variables.

The Cronbach's alpha values for usability, affect, and user value were 0.887, 0.743, and 0.728, respectively. In this study, all elements showed a Cronbach's alpha of 0.7 or higher, and the overall Cronbach's alpha was 0.917, indicating an overall high-reliability.

Among the 82 responding subjects, the majority were nursing officers (76.8%), with the remainder being chief nursing officers (11.0%), medical officers (3.7%), and others (8.5%). Considering rank, first lieutenants, second lieutenants, and captains were 62.2%, 1.2%, and 31.7%, respectively, of the total. The rest of the subjects were civilian personnel (4.9%). The age (mean ± standard deviation) was 27.23 ± 3.77 years, the mean length of time working at an armed forces hospital was 2.87 ± 1.696 years, the mean length of clinical experience was 3.10 ± 1.796 years, the mean length of time using N-DEMIS was 2.80 ± 1.543 years, and the mean length of time using systems other than N-DEMIS was 1.85 ± 1.895 years. Almost of all the subjects had undergraduate degrees as their highest educational attainment (93.9%). Others had 3-year undergraduate program college degrees (2.4%) or graduate or higher degrees (3.7%) for their highest educational attainment. The major employment type was compulsory military service (69.5%), and the rest of the subjects were in extended or long-term service (25.6%) or other types, representing contract employees, such as civilian personnel (4.9%).

Table 1 shows the means and standard deviations for each element of UX. Among the elements of UX, the mean scores for usability and affect were similar, at 3.11 and 3.10, respectively, while the mean score for user value was the lowest, at 3.00. Examination of the standard deviations shows that the distribution of scores across users was relatively even.

We performed a t-test to analyze the differences between the elements as seen in Figure 1. The p-values for usability-affect, user value-usability, and affect-user value were 0.905, 0.046, and 0.098, respectively, indicating a significant difference between user value and usability, and no significant differences between the other pairs.

We also tested for differences in UX depending on the subgroups for each of the measured subject characteristics, and the results are shown in Table 2. Continuous variables were divided into subgroups based on distribution, as shown in Table 2. We observed significant differences in UX in relation to length of time working in an armed forces hospital (p = 0.050) and employment type (p = 0.010). Length of time working in an armed forces hospital did not show a significant relationship in the correlation analysis (p =0.549), so there was no specific trend, but the UX scores tended to be lower for subjects who had between 2 and 5 years of experience. For employment type, the highest UX scores were shown by the compulsory military service group, while the lowest UX scores were shown by the other group. From the multiple comparison test, the compulsory military service group had a significant difference from the extended or long-term service group (p = 0.048); however, there was no significant difference from the other group (p = 0.104).

When each component was assessed in relation to subject characteristics, all three components of UX (usability, affect, and user value) differed significantly according to the employment type, and compulsory military service showed the highest scores for all three components, which is aligned with the UX results.

A multiple linear regression analysis with backward elimination was conducted, and the variables used are shown in Table 3. The ANOVA for regression showed that the model was significant at the 0.003 probability level, and R² was equal to 0.188. Among the highest educational attainments, only the 3-year undergraduate program college degree showed significance in the regression analysis. In addition to educational attainments, the length of time working in an armed forces hospital and employment types were selected as determinant factors for UX of N-DEMIS.