The aim of this study was to use discrete event simulation (DES) to model the impact of two universal suicide risk screening scenarios (emergency department [ED] and hospital-wide) on mean length of stay (LOS), wait times, and overflow of our secure patient care unit for patients being evaluated for a behavioral health complaint (BHC) in the ED of a large, academic children’s hospital.
We developed a conceptual model of BHC patient flow through the ED, incorporating anticipated system changes with both universal suicide risk screening scenarios. Retrospective site-specific patient tracking data from 2017 were used to generate model parameters and validate model output metrics with a random 50/50 split for derivation and validation data.
The model predicted small increases (less than 1 hour) in LOS and wait times for our BHC patients in both universal screening scenarios. However, the days per year in which the ED experienced secure unit overflow increased (existing system: 52.9 days; 95% CI, 51.5–54.3 days; ED: 94.4 days; 95% CI, 92.6–96.2 days; and hospital-wide: 276.9 days; 95% CI, 274.8–279.0 days).
The DES model predicted that implementation of either universal suicide risk screening scenario would not severely impact LOS or wait times for BHC patients in our ED. However, universal screening would greatly stress our existing ED capacity to care for BHC patients in secure, dedicated patient areas by creating more overflow.
Suicide is the second leading cause of death among youth aged 10–24 years [
Two different system interventions have been considered at our pediatric center to help address suicide risk: universal suicide risk screening for ED patients ≥10 years old and hospital-wide patients ≥12 years old. Our existing system and the interventions under consideration involved ED-based evaluation by behavioral health specialists in secure patient care areas for any child at imminent risk for suicide.
The ED includes mental health professionals and secure patient care areas to offer safe and emergent evaluations of patients at imminent risk of suicide. However, these resources are limited. Furthermore, patients with behavioral health complaints in our setting are already subjected to long stays in the ED. It was unknown how the implementation of universal screening would impact the availability of ED resources or throughput for patients who currently require evaluation for BHC.
(1) To test the hypothesis that universal suicide risk screening (ED and hospital-wide) would increase the mean length of stay (LOS) for patients with a BHC by more than 1 hour in a discrete event simulation model of the ED and (2) to compare system performance (LOS, wait times, and secure unit overflow) between models of the existing system and both proposed universal screening scenarios (ED and hospital-wide).
We used discrete event simulation (DES) modeling to build a computational representation of patient flow through the part of our ED where patients are treated for BHC. We sought to use DES to predict the system impact on ED patient flow from universal suicide risk screening, since DES is well-suited for testing theoretical changes to a complex system [
Patients 0–21 years old are treated in the study setting: a large, urban, free-standing, academic children’s hospital ED in the United States with approximately 90,000 annual visits.
We created a conceptual model representing the flow of patients who required evaluation by a behavioral health specialist in the ED. The flow of patients through the existing system is depicted in
In this scenario, all ED patients ≥10 years old, including those who did not present for a BHC, would be screened. If imminent suicide risk were detected, then it was assumed that these children would then be evaluated by a behavioral health specialist in the secure area in the ED.
In this scenario, all ED patients ≥10 years old and all patients hospital-wide ≥12 years old (regardless of presenting complaint), would undergo screening. Similar to the first scenario, patients who screened positive for imminent risk of suicide in any outpatient clinic would be referred to the ED for an emergent evaluation by a behavioral health specialist in the secure area.
Our team included local experts to ensure that the layout and general allocation of rooms and staff in the model accurately represented our ED. In order to model parameters for medical care, behavioral health evaluation, and disposition times, we utilized site-specific data from actual patients from January 1, 2017 to December 31, 2017. We randomly selected dates to create a 50/50 split of patient tracking data from 2017 for derivation and validation datasets. Our proposed universal screening scenarios include the Columbia-Suicide Severity Rating Scale (C-SSRS), one of five evidence-based screening tools recommended by the Joint Commission for suicide risk screening [
Our DES model was built using Simio (v10.165), using parametric probability distributions for all processes, fitted to the shape of retrospective data. The DES model represents how a patient who requires a behavioral health evaluation will flow through the ED, using probability distributions to approximate the real-world variability around the duration of each evaluation and disposition decision. Staffing was built around a standard week-long ED schedule and patient arrival patterns incorporated the variability around each hour of the day, day of the week, and month of the year.
Patient flow through the model (including the existing system and proposed interventions) was verified by our study team. Validation was performed by simulating 1 year of system flow at steady state, and repeating this with 1,000 iterations to compare model output predictions of LOS, wait times for behavioral specialist assessment, and secure unit overflow to actual site-specific data. The ED psychiatry leadership indicated they would be most interested in predictions from a model with at least 95% accuracy; therefore, we set an a priori validation cut-off of within 5% of historical data for the mean of each model output metric.
Model outputs, including LOS, wait times, and secure unit overflow, are dependent on complex interactions between patient needs and available resources, defined by model parameters. We used a validation dataset, also from site-specific records between January 1, 2017 and December 31, 2017, to compare against output metrics from our model built with parameters defined by a separate derivation dataset from the same year. Validation served as the final error-checking step to demonstrate that our model would produce output metrics that adequately represented the real-world system.
We simulated 1,000 iterations of 1 year of patients flowing through our model, incorporating the additional demands on behavioral health resources that we expected from the implementation of universal ED screening. We simulated another 1,000 iterations of 1 year, including changes to represent universal hospital-wide screening. For each scenario, we collected LOS, defined by ED arrival to disposition for BHC patients. We also collected wait times for behavioral health evaluation, defined by the time in queue after completing medical clearance, and the number of days each year with unit overflow, during which BHC patients exceeded the space available in secure patient areas (
In our first sensitivity analysis, we explored how the use of additional ED personnel resources would impact our results. In the existing system, our behavioral health specialists are psychiatry-specific social workers, who perform all behavioral health evaluations in the ED. The addition of brief safety assessments for screen-positive patients would represent additional clinical work for our behavioral health specialists. We performed a theoretical experiment in which other ED social workers completed all brief safety assessments for screen-positive patients not at imminent risk.
We performed additional sensitivity analyses by varying the proportion of positive screens, as our estimates were based on adult data and the screen-positive rate of adolescents for suicide risk might be different.
This project was undertaken as a quality improvement initiative at Children’s National and it did not constitute human subjects research. As such, it was deemed exempt from oversight of the Institutional Review Board.
Based on annual volumes at our center, we estimated that universal ED screening of non-BHC patients would detect approximately 684 ED patients per year requiring emergent evaluation by a behavioral health specialist. Universal hospital-wide screening would detect approximately 2,989 outpatients per year requiring emergent behavioral health evaluations, in addition to the patients detected with universal ED screening. Other model parameters, including parameters for the probability distribution used for each process, are reported in
Local stakeholders, including ED psychiatry leadership, reviewed our model and the anticipated patient flow for both proposed scenarios, verifying that patients moved from process to process appropriately. We validated the model, with mean LOS, wait times, and unit overflow within 5% of our validation dataset (
Patients with BHC in the ED had similar mean LOS in the model of the existing system (11.28 hours; 95% confidence interval [CI], 11.23–11.33 hours), universal ED screening (11.48 hours; 95% CI, 11.43–11.53 hours), and universal hospital-wide screening (11.88 hours; 95% CI, 11.82–11.94 hours). The number of days per year that BHC patients exceeded the space available in secure patient areas increased from 52.9 days (95% CI, 51.5–54.3 days) in the model of the existing system to 94.4 days (95% CI, 92.6–96.2 days) and 276.9 days (95% CI, 274.8–279.0 days) for universal ED screening and hospital-wide screening, respectively (
The predicted increase in unit overflow (to 93.5 days per year with universal ED screening; 95% CI, 91.9–95.1 days per year) was not mitigated by using ED social workers to relieve psychiatry-specific social workers from performing brief safety assessments (other metrics included in
We developed a theoretical model that predicts significant system impacts of implementing universal screening for suicide risk. We validated the DES model and compared screening scenarios using mean LOS, wait times, and secure unit overflow as output metrics. Based on this DES model, we expect minimal changes to LOS for BHC patients after implementation of either universal suicide risk screening scenario. However, our model suggests that universal suicide risk screening in the ED could nearly double the number of days each year that BHC patients will exceed the secure patient areas dedicated to this population.
Although this work is theoretical, we employed strategic elements that are considered central to improving experiences for real patients, including policies and measurement, quality, and innovation and technology [
The coronavirus disease 2019 (COVID-19) pandemic has been associated with increasing demand and decreasing access to mental health resources [
As our systems continue to face new challenges, such as increasing numbers of patients needing mental health services and limited resources, computer-based modeling is an economically friendly and useful tool. DES has been used for decades to support staffing and operational planning in hospitals [
Our simulation work was impactful in two ways. First, ED psychiatry leadership proceeded with a phased approach of suicide risk screening, in part because our model demonstrated the potential for drastic worsening of behavioral health patient overflow out of the secure patient area [
This study has several limitations. The most important limitation of our model is that we treated the wait time for inpatient beds as a model input independent of the number of patients requiring admission. This assumption, which allowed us to constrain our modeling efforts to the ED as a system, was appropriate for our clinical setting because we have the ability to transfer admitted patients to other regional facilities. If we were unable to transfer patients to outside facilities, our model outputs would underestimate the ED flow impact of suicide risk screening. For this reason, our findings may not be generalizable to ED settings without the ability to transfer patients to outside psychiatric facilities.
We assumed that 100% of ED patients ≥10 years old and all patients hospital-wide ≥12 years old would be eligible for screening. Acutely ill ED patients were included in the model because they can be screened after they are stabilized. However, we did not exclude patients with respiratory failure or those taken for emergency surgery, who would be ineligible for screening in the real world, potentially exaggerating the detrimental impact of universal screening in our model.
Our model may require modifications as we gather additional observational data from our site’s experience with the C-SSRS. We did not perform direct observations to confirm the duration of processes in our model. However, site-specific patient tracking data are routinely used for quality improvement and operational decisions at our center and a team of local experts reviewed our retrospectively derived process durations. Although the C-SSRS has been extensively studied in adults, we have not yet assessed test characteristics or the proportion of positive screens in our ED [
The operational planning for universal suicide risk screening was informed by these modeling efforts. For example, in our models, psychiatry social workers perform a brief safety assessment for every screen-positive ED patient, in addition to their complete evaluation for existing and screen-positive patients at imminent risk of suicide. Currently, ED social workers complete these evaluations. In preparation for universal ED screening, the ED psychiatry leadership has made other efforts to optimize resource allocation, including support for medical providers to better discern which patients require behavioral health evaluations, the addition of an ED psychiatrist to safely discharge patients with BHCs who might otherwise be hospitalized, and more expeditious transfer of patients when our inpatient service is at capacity. Additionally, some outpatient clinics (notably, the outpatient psychiatry clinics) have already begun documenting suicide risk screens at least once annually for older patients. Based on these ongoing, incremental efforts to prepare for and increase suicide risk screening, the real-world detrimental system impacts of implementing universal screening would likely be less pronounced than our model predictions.
The overall lesson from our experience is that DES modeling suggested that an unacceptable increase in unit overflow would take place in response to an abrupt implementation of universal suicide risk screening. Incremental increases in screening are critical to maintaining the situational readiness of our ED, and successful implementation of screening should be accompanied by aggressive strategies to decrease waiting time for hospitalization, reduce boarding times, and minimize unnecessary admissions.
No potential conflict of interest relevant to this article was reported.
This article was presented as a platform presentation at the Pediatric Academic Societies Meeting in Baltimore, MD, April 29, 2019.
Supplementary materials can be found via
Conceptual model for pediatric emergency department (ED) evaluations of patients with behavioral health complaints.
Emergency department (ED) layout, including the secure patient care area dedicated to patients with behavioral health complaints, adjacent patient care locations (“Area C”), and the decontamination area, which serves as the primary overflow when the number of patients with behavioral health complaints exceeds the space available in secure patient areas.
Simio measure of risk and error (SMORE) plot, showing the number of days each year with overflow of patients with behavioral health complaints exceeding the capacity of secure patient care areas. The model outputs for our existing system, as well as models with universal emergency department (ED) screening and universal hospital-wide screening, are represented as a box-and-whisker plot demonstrating the differences in expected unit overflow between these models. The means are presented as orange circles, 95% confidence intervals (CIs) around means are represented by beige bars, and 95% CIs around the 25th and 75th percentiles are represented by blue bars.
Model parameters
Process | Data source | Parameter | Service time distribution |
---|---|---|---|
Patients evaluated by psychiatry team | |||
Arrival rate |
Derivation dataset (n = 1,058) | Poisson (λ) | - |
Admission rate | Derivation dataset (n = 448 of 1,058) | Fixed 42.3% chance | - |
Boarding rate |
Derivation dataset (n = 85 of 1,058) | Fixed 0.08% chance | - |
Medical care for patients with behavioral health complaints | Derivation dataset (n = 1,049) | - | 9 + Weibull (155, 1.21) |
Complete behavioral health evaluation | Derivation dataset (n = 1,017) | - | 39 + Weibull (99.3, 1.08) |
Wait time for inpatient bed, admitted patients | Derivation dataset (n = 397) | - | 9 + Weibull (527, 0.77) |
Boarding time for emergency department (ED) boarders | Derivation dataset (n = 56) | - | 137 + Weibull (1220, 1.27) |
Nurse discharge process | Expert opinion | Triangular (5, 10, 15) |
Variable arrival rate with distinct λ each hour, ranging from 0.001 to 1.089 patients per hour depending on month, day of week, and hour of day.
Boarding rate includes only patients who board in the ED prior to discharge home. Patients that board in the ED prior to admission are included in admission rate.
For distribution Weibull (beta, alpha), beta is scale parameter, alpha is shape parameter.
The Weibull distribution has a closed-form inverse cumulative distribution function given by: F-1(U) = b [−ln(1-U)]1/a
To generate the Weibull distributions, random subset of n = 100 from derivation dataset was used for: medical care, complete behavioral health evaluation, and wait time.
Parameters for the Triangular distribution include (minimum, mode, maximum).
Model output performance against the ED validation dataset from 2017
Metrics for BHC patients | Validation dataset (n = 925) | Model output | Difference (%) |
---|---|---|---|
Wait time (hr) | 3.04 (2.37) | 2.91 ± 0.01 | −4.11 |
Length of stay (hr) | |||
Overall | 10.81 (10.94) | 11.28 ± 0.05 4.35 | |
Admitted patients | 15.58 (12.57) | 15.58 ± 0.09 | −0.01 |
Boarded patients | 25.91 (12.52) | 26.35 ± 0.24 | 1.69 |
Discharged patients | 5.22 (2.75) | 5.17 ± 0.02 | −0.96 |
Secure unit overflow (day/yr) | 52 (0) | 52.90 ± 1.40 | 1.73 |
Values are presented as mean (standard deviation) or mean ± half-width of 95% confidence interval. ED: emergency department, BHC: behavioral health complaint.
Outcome measures for models of the existing system and two proposed system changes
Metrics for BHC patients | Model output | ||
---|---|---|---|
Existing system | Universal ED screening | Universal hospital-wide screening | |
Wait time (hr) | 2.91 ± 0.01 | 3.09 ± 0.02 | 3.57 ± 0.03 |
Length of stay (hr) | |||
Overall | 11.28 ± 0.05 | 11.48 ± 0.05 | 11.88 ± 0.06 |
Admitted patients | 15.58 ± 0.09 | 15.76 ± 0.09 | 16.17 ± 0.09 |
Boarded patients | 26.35 ± 0.24 | 26.38 ± 0.24 | 26.88 ± 0.25 |
Discharged patients | 5.17 ± 0.02 | 5.33 ± 0.02 | 5.82 ± 0.03 |
Secure unit overflow (day/yr) | 52.90 ± 1.40 | 94.40 ± 1.80 | 276.90 ± 2.10 |
Values are presented as mean ± half-width of 95% confidence interval.
ED: emergency department, BHC: behavioral health complaint.
Outcome measures for models of the existing system and two proposed system changes, with support from social workers
Metrics for BHC patients | Model output | ||
---|---|---|---|
Existing system | Universal ED screening |
Universal hospital-wide screening | |
Wait time (hr) | 2.91 ± 0.01 | 3.08 ± 0.01 | 3.49 ± 0.03 |
Length of stay (hr) | |||
Overall | 11.28 ± 0.05 | 11.39 ± 0.05 | 11.85 ± 0.06 |
Admitted patients | 15.58 ± 0.09 | 15.64 ± 0.09 | 16.16 ± 0.08 |
Boarded patients | 26.38 ± 0.24 | 26.54 ± 0.24 | 26.95 ± 0.24 |
Discharged patients | 5.17 ± 0.02 | 5.33 ± 0.02 | 5.75 ± 0.03 |
Unit overflow (day/yr) | 52.90 ± 1.43 | 93.50 ± 1.60 | 266.00 ± 2.02 |
Values are presented as mean ± half-width of 95% confidence interval.
BHC: behavioral health complaint.
Social worker performs brief safety assessment. In our initial testing of proposed system changes, we programmed our model to have brief safety assessments performed by psychiatry social workers, responsible for all other behavioral health complaint management in the emergency department. Subsequent pre-implementation planning has proposed the use of emergency department social workers for these brief safety assessments to make sure psychiatry social workers remain available for complete behavioral health evaluations.