Triage Data-Driven Prediction Models for Hospital Admission of Emergency Department Patients: A Systematic Review
Article information
Abstract
Objectives
Emergency department (ED) overcrowding significantly impacts healthcare efficiency, safety, and resource management. Predictive models that utilize triage information can streamline the admission process. This review evaluates existing hospital admission prediction models that have been developed or validated using triage data for adult ED patients.
Methods
A systematic search of PubMed, Embase, CINAHL, Web of Science, and the Cochrane Library was conducted. Studies were selected if they developed or validated predictive models for hospital admission using triage data from adult ED patients. Data extraction adhered to the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), and the risk of bias was evaluated using PROBAST (Prediction model Risk of Bias Assessment Tool).
Results
Twenty studies met the inclusion criteria, employing logistic regression and machine learning techniques. Logistic regression was noted for its traditional use and clinical interpretability, whereas machine learning provided enhanced flexibility and potential for better predictive accuracy. Common predictors included patient demographics, triage category, vital signs, and mode of arrival. The area under the curve values for model performance ranged from 0.80 to 0.89, demonstrating strong discriminatory ability. However, external validation was limited, and there was variability in outcome definitions and model generalizability.
Conclusions
Predictive models based on triage data show promise in supporting ED operations by facilitating early predictions of hospital admissions, which could help decrease boarding times and enhance patient flow. Further research is necessary to validate these models in various settings to confirm their applicability and reliability.
I. Introduction
Emergency departments (EDs) play a critical role in delivering timely and appropriate emergency medical services to patients with acute illnesses of varying severity, often under unpredictable conditions. However, ED overcrowding is an increasing global problem, resulting in extended waiting times, inefficient resource utilization, and compromised patient safety [1]. To tackle these issues, healthcare systems need effective resource allocation strategies and informatics-driven solutions that facilitate prompt clinical decision-making. Specifically, predictive models that utilize real-time data and advanced algorithms can greatly improve the operational efficiency of EDs by accurately determining which patients need hospital admission [2,3]. In the current system, the admission process starts only after a patient's ED visit is complete, causing prolonged waiting times and unnecessary overcrowding as patients wait for administrative staff to process their admission and assign hospital beds. By implementing early prediction models at the time of patient arrival, the administrative process could be initiated simultaneously with the ED visit, thus reducing waiting times and preventing further ED crowding if the patient requires admission [4]. This approach would streamline patient flow and improve overall ED efficiency [2].
Predictive models have been developed and implemented to address these challenges, with the goal of forecasting which ED patients will require hospital admission [5,6]. By accurately predicting the need for admission early in the triage process, these models can reduce boarding times, improve resource allocation, and improve the overall efficiency of the ED [4].
Triage information, which includes age, sex, vital signs, and mode of arrival, is collected during the initial patient assessment in the ED. These data are crucial for quickly evaluating patient conditions and can be used to develop predictive models for hospital admission [7–9]. However, the systematic development of predictive models based solely on triage data is limited. This limitation is due to the reliance on detailed clinical data, such as laboratory or imaging results, which are not available at triage. Additionally, variability in triage systems and practices across different institutions hinders the generalizability of these models [10–12]. Previous reviews have highlighted usability challenges in clinical implementation [10] and the potential of machine learning (ML) techniques to improve prediction accuracy [12], laying the groundwork for further research into models that leverage triage-only data. Models based on triage data are broadly applicable across EDs, aiding in the evaluation of patient conditions and informing admission decisions. Advances in artificial intelligence and ML techniques have further improved predictive accuracy, enabling the development of sophisticated triage-based models [13].
This systematic review aims to identify and assess studies that have developed or validated hospital admission prediction models for adult ED patients using triage data. It specifically targets adult patients, acknowledging the unique clinical characteristics and care pathways that differ from pediatric populations. By examining the features and limitations of these models, this review intends to offer insights into enhancing ED resource management and the quality of patient care, as well as propose directions for future research.
II. Methods
1. Study Design
This study is a systematic review of studies on hospital admission prediction models using ED triage data and was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [14].
2. Eligibility Criteria
This study included all studies that developed or validated predictive models for the hospital admission of ED patients. It included data collected during the triage stage and utilized both retrospective and prospective study designs.
Inclusion criteria:
1) Participants: Adult patients aged 16 years or older presenting to the ED. Studies involving mixed-age populations were also included, provided their findings were stratified by age groups or were relevant to adults.
2) Intervention: Predictive models developed using data collected during the triage stage.
3) Outcomes: Hospital admission, defined as including both general ward and intensive care unit admission.
4) Only peer-reviewed journal articles were included.
Exclusion criteria:
1) Studies that focused exclusively on pediatric populations or specific diseases or symptoms.
2) Studies that were published in languages other than English or Korean.
3. Data Sources and Search Strategy
The literature search was conducted on October 20, 2023, utilizing five databases: PubMed, Embase, CINAHL, Web of Science, and the Cochrane Library, with no restrictions on publication dates. Additionally, manual checks of the references from the retrieved studies were performed to identify additional studies that met the inclusion criteria.
Search terms were constructed using MeSH terms, and queries were adapted to the specific features of each database. The search terms were combined using the operators OR and AND. The primary MeSH terms employed were “emergencies,” “triage,” and “prognosis.” The final search strategy was formulated as follows: (emergency OR emergencies OR emergence OR emergent OR emergencies[MeSH Terms]) AND (triage OR triages OR triaging OR triaged OR triage[MeSH Terms]) AND (prognosis OR prediction OR predictive OR predicting OR predict OR prognosis[MeSH Terms]) AND (model OR modeling OR tool).
4. Study Selection
The references retrieved were organized using reference management software (EndNote 20.6; Clarivate, Philadelphia, PA, USA). After reviewing the titles, duplicates were removed. Titles and abstracts were further screened, and studies that were clearly irrelevant to the research question were excluded. The study selection process was conducted independently by two researchers based on the core research question, and inclusion and exclusion criteria. In cases of disagreement, a third researcher facilitated consensus to finalize the selection of studies.
5. Data Extraction
Data extraction from the selected studies was independently conducted by two researchers and subsequently verified. In cases of disagreement, a consensus was reached through discussion with a third researcher. The data extraction form was based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) [15]. The extracted data encompassed authors, publication year, country, setting, type of prediction modeling, data source, study design, study period, population, sample size, outcome variable, candidate predictors, important variables, handling of missing data, algorithms used, validation methods, and the performance of the final model.
6. Risk of Bias and Applicability Assessment
The risk of bias and applicability of each study were independently assessed by two researchers using the Prediction model Risk of Bias Assessment Tool (PROBAST) checklist [16]. Any disagreements were resolved through consultation with a third researcher. PROBAST systematically evaluates the risk of bias (ROB) by examining four critical domains—participants, predictors, outcomes, and analysis—through 20 targeted questions that identify methodological biases. Applicability is determined by evaluating how well the study's population, predictors, and outcomes align with the research question, thus assessing the relevance and generalizability of the study findings.
III. Results
1. Study Selection
A comprehensive search across five databases identified a total of 3,690 records. After removing duplicates, 2,219 unique records were left. These underwent a screening process based on their titles and abstracts, which led to the exclusion of 1,819 records that did not relate to the research question. The abstracts of the remaining 400 records were further reviewed, and 76 studies that met the inclusion criteria were selected. Two independent reviewers then conducted a full-text assessment of these studies, resulting in the exclusion of 56 studies for various reasons, including the use of factor analysis instead of predictive modeling, a focus on non-adult populations, or the absence of relevant prediction outcomes. As a result, 20 studies were included in the final systematic review. The search process, based on the PRISMA 2020 flow diagram, is illustrated in Figure 1.
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart of study selection process.
2. Risk of Bias and Applicability
Using the PROBAST tool, most studies were assessed as having a low risk of bias in participant selection and outcome measurement. However, they encountered challenges with missing data and predictor selection. As shown in Table 1, although most studies conducted internal validation (e.g., cross-validation), only three studies utilized external validation methods [5,6,17], underscoring the need for enhanced generalizability of the models.
PROBAST results
3. Study Characteristics
The characteristics of the studies included in this review are summarized in Table 2. This systematic review included 20 studies [2–6,8–9,17–29], with research settings distributed across various countries and regions, including the USA, Australia, the UK, Spain, Singapore, Taiwan, and Austria. The methodologies predominantly involved retrospective analyses, although several studies also featured prospective validation [5,6,17]. While most studies focused exclusively on model development, a few integrated both development and validation processes [17,18]. The data sources were diverse, ranging from ED databases and hospital information systems to national health surveys, such as the National Hospital Ambulatory Medical Care Survey. These studies were conducted in both single-center and multicenter environments, with some specifically employing electronic health intelligence systems (eHINTS) or extensive datasets like the Medical Information Mart for Intensive Care – Emergency Department (MIMIC-IV-ED) [8,19]. The duration of the studies varied from 1 to 9 years, tailored to the particular design and scope of each study. The primary focus was on adult patients, though some studies included participants of all ages. Sample sizes varied widely, from 894 to over 3 million patients or events, reflecting the diverse scopes and settings of the studies.
Characteristics of the included studies
Table 3 summarizes the primary outcome assessed across the studies, which was hospital admission, encompassing both ward and intensive care unit (ICU) admissions. Other outcomes evaluated included mortality, critical outcomes, length of stay, and readmission [5,19–22]. Admission rates varied widely, ranging from 11.0% [2] to 47.3% [19], depending on the definition of admission, study setting, and patient population.
Summary of the included studies
The studies reviewed incorporated a variety of candidate predictors, including patient demographics (such as age, sex/gender, and ethnicity), triage details (such as triage category, vital signs, chief complaints, and mode of arrival), medical history (including previous admissions and comorbidities), and administrative information (such as date and time of attendance, shift time, and insurance status). Key predictive variables that were often identified included age, triage category, mode of arrival (for example, ambulance), and vital signs like body temperature, heart rate, and respiratory rate.
Regarding data handling, several studies addressed the issue of missing data either by excluding incomplete cases or by employing imputation techniques. For instance, Xie et al. [19] utilized median imputation, whereas Cameron et al. [23] applied a combination of exclusion, removal, and imputation strategies. However, despite being retrospective in nature, four studies failed to provide details on how they managed missing data [9,18,24,25].
Various ML algorithms were utilized in the studies reviewed, with logistic regression being the predominant method. More recent research has incorporated advanced algorithms, including gradient boosting machines (GBM), random forest, and neural networks, as well as more complex models like long short-term memory [2,19,21,22,26].
Validation methods varied across the studies. Many employed cross-validation techniques, such as bootstrap cross-validation [3,20,23] and k-fold cross-validation [2,4,26]. Several studies also conducted external validation [5,6,17], which enhances the generalizability of their findings.
4. Model Performance of the Final Models
The study compared the performance of various predictive models designed to forecast hospital admissions using triage data from the ED, as detailed in Table 4. The evaluation concentrated on three key aspects: discrimination, calibration, and classification. Each aspect was crucial for assessing the primary outcome, which was hospital admission.
Model performances of the final model
Discrimination, primarily assessed through the area under the receiver operating characteristic curve (AUC), indicated how well each model could differentiate between patients who required hospital admission and those who did not. The GBM model developed by Cusido et al. [2] exhibited the highest discrimination ability, achieving an AUC of 0.891 (95% CI, 0.890–0.892). This was closely followed by the Glasgow Admission Prediction Score (GAPS) model by Cameron et al. [23], which recorded an AUC of 0.877. The Sydney Triage to Admission Risk Tool (START), introduced by Dinh et al. [3], demonstrated an AUC of 0.820. Although slightly lower than the previous models, it still showed robust discrimination performance.
In terms of calibration, the Hosmer-Lemeshow goodness of fit test was commonly used to evaluate the alignment between predicted probabilities and actual outcomes. The GAPS model [23], with a p-value of 0.524, demonstrated strong calibration, indicating that its predictions closely matched observed admissions. In contrast, the START model [3] exhibited poor calibration (p<0.001), suggesting that despite reasonable discrimination, there was a significant mismatch between its predictions and the actual outcomes.
When assessing classification performance, the accuracy metric exhibited significant variations across the models. Cusido et al. [2] achieved the highest accuracy (89.8%), while Cameron et al. [23] followed with 80.3%. The positive predictive value and negative predictive value were evaluated in seven studies [3,6,8,9,21,22,28], indicating the models' effectiveness in accurately predicting true positives and true negatives. Additional metrics, such as the F1-score [22] and net reclassification improvement [21,24], provided further insights into the classification capabilities and enhancements of these models.
IV. Discussion
This systematic review identified a total of 20 studies that focused on developing predictive models for hospital admissions in ED settings. Of these, 16 models were newly developed, including START [3] and GAPS [23]. Additionally, four studies were dedicated to expanding or externally validating the START [5,6,17] and GAPS [26] models. These efforts enhanced the clinical applicability of both models. The majority of the studies employed logistic regression, valued for its simplicity and interpretability, as the primary algorithm. However, more recent studies have shifted towards ML and deep learning techniques to improve predictive accuracy. Despite the potential advantages of deep learning models, such as their ability to capture complex patterns in data, several studies noted that deep learning approaches often required substantial computing resources and time without offering significant improvements in model performance over traditional ML methods [26]. This underscores an important consideration for practical implementation, where computational efficiency is often as crucial as predictive accuracy, especially in time-sensitive environments like the ED.
The operational definition of hospital admission, which is the primary outcome variable in this review, varied across the included studies. Some studies counted transfers to other hospitals as part of the admission outcome [3,4,21,26], while another study included patients who died in the ED [23]. Other studies categorized hospital stays into different durations based on clinical objectives [3,6,17,26]. For the purposes of this review, hospital admission is defined to include both general ward and intensive care unit admissions. This definition supports the review's objective of predicting admissions during triage, which helps reduce boarding times and optimize hospital capacity management, thereby ensuring a comprehensive evaluation of admission outcomes.
The predictors identified as important variables in predictive models were largely consistent across studies, including age, sex, vital signs, and mode of arrival, which are among the most commonly used. These variables are routinely collected in the ED. Triage-based models, which do not require additional or complex data collection, provide immediate predictions that significantly contribute to timely decision-making in an emergency setting. In contrast, models that utilize laboratory or imaging data leverage detailed clinical information to achieve higher accuracy [4,21]. However, they depend on data that are not available during the triage stage. This distinction further underscores the practical value of triage-based models in real-world clinical applications.
Despite the generally high performance of the models, as indicated by AUC values typically ranging from 0.80 to 0.89, several studies have noted challenges related to missing data and varying data quality across different hospital settings [30]. While some models excelled in discrimination, others showed stronger calibration or classification performance. The models by Cusido et al. [2] and Cameron et al. [23] were particularly notable as top performers, providing a balance of high discrimination and accurate classification, making them promising tools for predicting hospital admissions from ED triage data.
Although internal validation was performed in most studies using methods like cross-validation or bootstrapping, external validation was conducted in only a few instances. This raises concerns about the generalizability of these models across various clinical contexts.
Our findings are consistent with previous systematic reviews, such as the one conducted by Brink et al. [10], which evaluated admission prediction models but noted their limited real-world application due to challenges in clinical usability and validation. Brink's study was confined to European countries, which restricts its global applicability. In contrast, our review encompasses studies from a broader range of countries, enhancing its relevance to a wider variety of clinical settings. Sanchez-Salmeron et al. [12] proposed that ML-based models hold promise as effective tools for enhancing triage-based predictions. However, the deployment of these models in EDs faces significant hurdles, especially the substantial computational resources needed for real-time predictions.
This study makes a significant contribution to the ongoing efforts to refine hospital admission prediction models by specifically focusing on those that utilize critical triage information, which is readily available upon patient arrival. By systematically reviewing the performance and applicability of these models, we offer a comprehensive evaluation of the current state of hospital admission prediction tools. This review highlights the strengths and weaknesses of various algorithms and methodologies.
The review highlights the potential of models based on triage information to enhance ED operations, especially by facilitating earlier predictions of hospital admissions. This could lead to shorter boarding times and improved patient flow. Additionally, the inclusion of studies from various healthcare systems in the review broadens the applicability of its findings, providing insights relevant to diverse clinical settings.
Despite its contributions, this review has several limitations. First, the diversity in study designs, predictors, and outcome definitions complicates direct comparisons of model performance across studies. Additionally, although many studies addressed the issue of missing data, some did not disclose their data handling strategies, which could bias their results. Future research should enhance transparency in data handling and consider incorporating unstructured data, such as nursing assessments, which have been shown to improve predictive power in certain studies. Furthermore, the absence of external validation in most studies raises concerns about the generalizability of these models to different clinical settings. Future studies should focus on validating models across various EDs to confirm their wider applicability.
This review underscores the potential of predictive models for hospital admissions based on triage data in EDs. Models like START and GAPS, which have been subjected to both extension and external validation, are particularly promising for clinical implementation. Given the broad availability of the identified predictors in ED settings, these models show great promise in reducing boarding times and enhancing patient flow through earlier bed assignments.
Notes
Conflict of Interest
Mona Choi is an editorial member of Healthcare Informatics Research; however, she did not involve in the peer reviewer selection, evaluation, and decision process of this article. Otherwise, no potential conflict of interest relevant to this article was reported.