This study was a cross-sectional study using data from a single emergency department of a tertiary university hospital with an annual census of 60,000 patients. Retrospective data of all visits by patients to the adult emergency department from November 2016 to June 2019 were included in the study. Encounters with missing data were excluded. All data processing was done with Python and related scientific libraries, pandas, and scikit-learn [1718]. This study and its protocol were approved by the Institutional Review Board of Seoul National University Hospital (No. 1910-071-1070) with a waiver of informed consent. No personally identifiable data was part of the dataset. Data used in this study was retrieved from the clinical data warehouse of Seoul National University Hospital Patients Research Environment (SUPREME).

Only variables collected during triage were used in the analysis. The data were divided into three separate datasets: structured data only, text of nursing triage notes only, and both structured data and nursing triage notes. The structured data included gender, age, date and time of arrival, chief complaint, route to the emergency department, pain location and intensity, vital signs (blood pressure, heart rate, respiratory rate, oxygen saturation), and level of consciousness. The chief complaint was selected from a set of 607 codes; the route taken by the patient to the emergency department was selected from five options (direct, refer from outpatient department, transfer from other hospital, other, or unknown); the pain location was selected from 10 possible locations; level of consciousness was selected from five options (alert, verbal response-disoriented, verbal responsestuporous, response to pain, and unresponsive). Some additional preprocessing was applied to the data. Arrival time was binned into 1-hour increments. Pain intensity scores that were written as text were converted into numeric form, for instance ‘6–7' was converted to 6.5. The nursing triage note is a free-form text note, usually one to three sentences in length that summarizes the patient's reason for visiting the emergency department written by the triage nurse. The outcome variable was the KTAS level assigned to the patient by the triage nurse on duty.

Natural language processing (NLP) is a set of methods for analyzing human languages. Its purpose is to allow computers to process and understand the meaning of text. Because each language has specialized structures, such as characters, grammar, and words, NLP techniques have been developed for each specific language. In this study, most of the targeted documents were written in Korean. The domain of documents should also be considered when implementing NLP. The targeted documents were triage notes containing both Korean and English words. Most of them followed Korean grammar but had many English words designating medical concepts, such as symptoms, medications, and disease names.

To analyze the triage notes, we used soynlp, which is an NLP library of unsupervised NLP techniques written and available in Python [19]. Soynlp was used to tokenize the triage notes and normalize words. Unlike Western languages like English, Korean is not divided into meaning structures by spaces alone. For this reason, most Korean morpheme analyzer tools are based on pre-defined dictionaries. However, it is very difficult for these tools to analyze text that contain words that are not in the dictionary. This is called an out-of-vocabulary problem. In contrast, soynlp uses a word extractor based on probabilistic scores representing the likelihood of words. Therefore, it is more appropriate for analyzing documents with terminology that is used only in specific domains compared to any other Korean processing tools.

Because of the time pressure involved in writing the initial triage note, they usually contain abbreviations and shorthand notations. Because punctuation marks and numbers are usually removed in the preprocessing step before analysis, to maintain the meaning of notations that include multiple punctuation marks, the most common shorthand notations were fully spelled out. For example, abbreviations such as ‘n/v’ were replaced with ‘nausea vomiting’ and notations such as ‘(+/−)’, which would be completely removed by the preprocessing step removing punctuation, was replaced with the words ‘positive negative’. There was a total of 165 such replacement rules. After these abbreviations and shorthand notations had been removed, punctuation and numbers were removed, and all English text was converted into lowercase. No special preprocessing of the Korean language was performed besides tokenization with soynlp.

In this study, we used logistic regression, random forest, and XGBoost to classify encounters. We applied a ridge logistic regression which uses L2 penalty to avoid multicollinearity and overfitting. Random forest is an ensemble model to combine predicted results of multiple decision trees [20]. In the process of learning a random forest model, each decision tree trains on a random sample of the training dataset with replacement, and a random subset of features is selected at each candidate split of trees. Through these methods, the trained trees become mostly uncorrelated. Averaging the results of less correlated models has the effect of reducing the high variance of each model's performance. As a result, the random forest has improved generalization performance. Random forest has been used by many scientists due to its performance; it was the best model for even the most comprehensive benchmarking studies [21]. XGBoost is an advanced model of gradient boosting machines (GBM), which trains many decision trees in a sequential and additive manner [22]. Although both GBM and random forest are based on decision trees, they train trees in different ways. In random forest, each tree is trained independently and becomes a so-called fully-grown tree, which overfits a subset of training dataset. GBM, on the other hand, learns small trees which may underfit the training set. However, trees in GBM are trained sequentially, and each tree is trained in consideration of the errors of other tree models in the previous steps. In this way, as more trees are trained, the error of GBM is reduced. XGBoost makes GBM's training scalable by distributed learning with several approximation methods to parallelize training. Since XGBoost was proposed, it has outperformed other algorithms in many studies and competitions [23].

To apply these algorithms, we structured the clinical notes using a bag-of-words model. In a bag-of-words model, each document is represented as a vector that has the same length as the size of the vocabulary [24]. In this study, the vocabulary was composed of unigrams, bigrams, and trigrams appearing in more than 0.1% in the entire document set [25]. However, in preliminary experiments, the use of unigrams alone yielded better performance. Using bigrams and trigrams caused the document vector represented by the bag-of-words model to become too long, leading the machine learning algorithms to fall victim to the curse of dimensionality.

Each of the models was trained and evaluated on three separate datasets: structured data only, text of the nursing triage notes only, and both structured data and nursing triage notes. Precision, recall, F1-score, and area under the receiver operating characteristic (AUROC) curve with 95% confidence interval (CI) are reported.