PDF
Citation
Print
INTRODUCTION
Acute deterioration of hospitalized patients in general wards poses a significant challenge, often resulting in unplanned transfers to intensive care units (ICUs), initiation of cardiopulmonary resuscitation (CPR), or even death before ICU transfer. These major adverse events (MAEs) increase mortality and impose substantial financial burdens on both patients and healthcare systems [1-3]. Previous national surveys indicated that 14%–28% of ICU admissions were unplanned transfers from general wards rather than originating from the emergency department (ED) or operating room [4]. Moreover, patients transferred to the ICU from general wards exhibit higher mortality rates than those admitted directly from the ED or following surgery [3].
The factors contributing to acute deterioration are complex and multifaceted, hindering early prediction and intervention; however, many MAEs are preceded by detectable physiological derangements within the previous 24 hours [5,6]. While some such events are inevitable owing to underlying conditions, many are preventable through closer monitoring and timely medical intervention. Observational studies suggest that early intervention can prevent approximately 44% of in-hospital adverse events and 15% of unplanned ICU transfers (UITs) [4,6].
The rapid response system (RRS) was established in the 1990s [7] to enhance surveillance and proactive interventions aimed at preventing deterioration of patients admitted to the general wards. Subsequently, various early warning scores (EWSs) were developed and validated for use together with the RRS when monitoring patient physiological signs and predicting impending adverse events. Although these scoring systems have demonstrated promise [8], their reliance on generic thresholds often fails to account for individual patient variability, resulting in suboptimal predictive performance [9,10]. In a randomized controlled trial by Haegdorens et al. [11], an RRS utilizing the National Early Warning Score (NEWS) did not significantly reduce the incidence of UITs, cardiac arrest, or unexpected death. Furthermore, concerns have been raised regarding the high false-alarm rates generated by these systems, which can disrupt the RRS workflow [9,12,13].
To address these limitations, there is growing interest in incorporating artificial intelligence (AI) technologies into EWSs [14]. The use of AI-based algorithms in EWS may improve predictive accuracy by analyzing large volumes of clinical data and detecting subtle patterns often overlooked by conventional rule-based methods [15]. These advanced systems offer tailored monitoring by adapting to the unique profiles of individual patients, potentially delivering more precise and timely alerts [16]. However, the performance and impact of AI-based EWSs in real-world clinical settings require validation in prospective clinical studies [17].
We developed an AI-based clinical decision support system, called the VitalCare - Major Adverse Event Score (VC-MAES), to predict in-hospital MAEs, including UITs, cardiac arrest, or death among patients admitted to the general wards. The VC-MAES is generated in real time by analyzing structured data from electronic health records (EHRs), facilitating prompt identification of and intervention for patients at high risk of clinical deterioration.
This study aimed to externally validate the performance of VC-MAES by prospectively collecting real-world medical data. Additionally, we sought to compare its predictive accuracy with those of two widely used and extensively studied traditional EWSs, the NEWS and Modified Early Warning Score (MEWS).
MATERIALS AND METHODS
The study adhered to the ethical guidelines of the 1975 Declaration of Helsinki and was approved by the Institutional Review Board of Keimyung University School of Medicine (No. 2022-12-081), which waived the requirement for informed consent due to the retrospective nature of this study. The study was registered with the Clinical Research Information Service (CRIS) operated by the National Institute of Health under the Korea Disease Control and Prevention Agency (CRIS Registration No. KCT0008466).
Study Design and Setting
This prospective observational external validation study was conducted at Keimyung University Dongsan Hospital in Republic of Korea. The model was implemented in six medical-surgical general wards across two departments: internal medicine (IM) and obstetrics and gynecology (OBGYN). These two departments were selected as our primary aim was to validate the model's performance across markedly different patient populations, rather than simply splitting cases into medical versus surgical categories. Additionally, we sought to validate our AI-based EWS in OBGYN populations that are often underrepresented in such studies. This was a non-interventional study in which the model generated real-time predictions that were neither disclosed to healthcare providers nor used to guide clinical decision-making. The study period was from June 3, 2023, to January 31, 2024, and patients were followed until discharge or April 30, 2024, whichever occurred first.
Patient Selection
Patients were eligible for inclusion if they met the following criteria: (1) 19 years of age or older and (2) had five initial vital signs (systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, and body temperature) recorded in their EHR during hospitalization. Patients were excluded if they had been admitted to the labor and delivery unit or directly to the ICU from the ED or operating room. All other patients, including those with do-not-resuscitate (DNR) orders, were included.
Data Collection
Patient demographics and clinical information, including vital signs, chief complaints, admission and final diagnoses, code status, start and end times of surgery, medication orders, and laboratory data, were extracted from the EHR. Data addressing ICU admission and discharge times, time of death, and CPR initiation and termination times were also retrieved.
Outcomes and Objectives
The primary outcomes of interest were UITs, cardiac arrest, and death. Cardiac arrest was defined as the initiation of chest compressions, defibrillation, or both, as documented in the EHR [18]. We defined UITs as unplanned when they occurred unexpectedly from general wards, as opposed to pre-arranged events, such as scheduled postoperative admissions or direct transfers from the ED [11,19].
Algorithm: VC-MAES
The VC-MAES is a proprietary predictive system that applies a deep-learning approach to time-series clinical data. Building on a bidirectional long short-term memory (biLSTM) architecture, this binary classification model estimated the likelihood of MAEs in general ward patients 6 hours in advance. The VC-MAES uses two categories of input data: (1) dynamic features derived from hourly time-series data, including vital signs and laboratory test results, and (2) static features. While the biLSTM component processes time-series inputs, the static features are handled by the fully connected layers. The outputs from both networks were subsequently merged and passed through additional classification layers to obtain the final prediction. A schematic of the VC-MAES model architecture is shown in Supplementary Figure 1, and additional information regarding the classification model can be found in Sung et al. [20].
The model was trained using the entire inpatient data, comprising 334,185 hospitalizations of 209,825 adult patients between 2013 and 2017 at Severance Hospital, a 2,454-bed tertiary academic medical center in Seoul, Republic of Korea. The dataset included patients from more than 35 medical and surgical specialties. The model primarily uses five vital signs and patient ages to generate a VC-MAES ranging from 0 to 100, with higher scores indicating a greater risk of MAEs within the next 6 hours. When used prospectively, the score is updated whenever a new input feature is recorded in the EHR, ensuring that it reflects the most recent data. If available, the model can also incorporate optional variables, such as oxygen saturation, Glasgow Coma Scale (GCS), total bilirubin, lactate, creatinine, platelets, pH, sodium, potassium, hematocrit, white blood cell count, bicarbonate, and C-reactive protein, to provide a more comprehensive risk score.
The VC-MAES imputes missing values using the last-observation-carried-forward (LOCF) method. In this study, if no previous values were available, normal values were assigned (Supplementary Table 1). Because mental status assessments were only performed in the ICUs and not in the general wards at this institution, GCS scores were assigned a value of 15 to calculate the VC-MAES, following the model’s missing value imputation method. Similarly, for the NEWS and MEWS calculations, a score of 0 was assigned for the Alert, Voice, Pain, Unresponsive (AVPU) scale, corresponding to an “alert” level of consciousness.
Performance Evaluation and Statistical Analysis
We compared the demographic characteristics of patients in the non-event and event groups using chi-square tests for categorical variables and t-tests or Wilcoxon rank-sum tests for continuous variables, as applicable. The overall accuracy of the VC-MAES in predicting MAEs at different time intervals was assessed using the area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) analyses. The AUPRC analysis was conducted because our dataset was highly imbalanced and focused on positive cases, measuring precision and recall. A larger AUPRC indicates better performance in detecting the positive class, which is crucial in imbalanced datasets aiming to identify rare events. A bootstrap approach with 1,000 replicate samples drawn with replacement was employed to compare the AUROC and AUPRC values of the predictive models, forming empirical distributions to calculate confidence intervals and derive P-values from the differences in performance metrics. Additionally, to evaluate the threshold-based performance for each relevant NEWS and MEWS cutoff, we calculated the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and number needed to evaluate (NNE). These metrics were used to benchmark the performance of the established models against VC-MAES. Last, we compared the performance of the EWSs while excluding DNR patients to avoid selection bias due to markedly different patient demographics, characteristics, and/or care.
In addition to the time-point analyses, we assessed the model using the patient episode definition [21], in which each hospital admission was treated as a single comprehensive case that drew on one or more measurements of vital signs and other clinical parameters obtained throughout the hospital stay. Specifically, the first recorded VC-MAES, NEWS, and MEWS results upon admission were used, and binary logistic regression was performed to examine the association between these EWSs and both in-hospital MAEs. Furthermore, we investigated whether the initial scores were associated with prolonged length of stay (pLOS) to explore broader clinical utility beyond early deterioration prediction, such as resource allocation and guiding early discharge planning. We defined pLOS as the 75th-percentile length-of-stay threshold for the entire cohort [22,23]. Demographic variables that were significantly associated with MAEs and pLOS in the univariate analyses were included as covariates in the multivariate logistic regression model. All statistical analyses were conducted using R (version 4.4.0) and Python (version 3.11.8). The pROC package in R was used for ROC curve analysis. Statistical significance was set at P<0.05.
RESULTS
Baseline Characteristics
This study initially included 6,478 screened encounters, comprising 4,846 unique patients admitted to the general ward. After applying the exclusion criteria, 439 cases were removed: 423 were directly admitted to the ICU from the ED or operating room, and 16 had insufficient data for score calculation. A flowchart showing the patient selection procedures is provided in Figure 1. Ultimately, 6,039 events (4,447 patients) with data for 272,493 time points were included in the analyses. Of these, 217 encounters (3.6%) experienced MAEs, including 102 UITs, 13 cardiac arrests, and 102 deaths. The majority of MAEs occurred in the IM subgroup, with 207 of 2,177 cases (9.5%) experiencing MAEs. In contrast, only 10 of the 3,862 OBGYN cases (0.26%), including 1,568 obstetric cases, experienced MAEs.
Table 1 summarizes the baseline demographic characteristics and laboratory values of the event and non-event groups. The median age of the study population was 52.0 years. Patients who experienced MAEs had a significantly higher median age of 76.0 years compared to those without MAEs (51.0 years, P<0.001). The sex distribution revealed that 78.90% of the participants were female, largely because nearly 64% of the study population was drawn from the OBGYN department. At admission, VC-MAES, NEWS, and MEWS results were significantly elevated in the MAE group, reflecting more severe clinical presentations (VC-MAES, 6.2 vs. 1.4; NEWS, 3.0 vs. 1.0; and MEWS, 2.0 vs. 1.0; all P<0.001).
In a subanalysis stratified by specialty, the IM subgroup (n=2,177) had a median age of 70.0 years, which was older than that of the overall cohort. Among IM patients who experienced MAEs (n=207), the median age increased further to 78.0 years (P<0.001). Additionally, IM patients with MAEs exhibited significantly elevated baseline VC-MAES, NEWS, and MEWS compared to IM patients without MAEs (VC-MAES, 6.3 vs. 3.9; NEWS, 3.0 vs. 1.0; and MEWS, 2.0 vs. 1.0; all P<0.001).
In contrast, the OBGYN subgroup (n=3,862) consisted predominantly of younger females (median age, 41.0 years), with only 10 MAEs reported. Those who experienced MAEs were significantly older (60.5 years) than their counterparts without MAEs and also presented with significantly higher VC-MAES, NEWS, and MEWS at admission (mean VC-MAES, 4.9 vs. 0.9; NEWS, 2.5 vs.1.0; MEWS, 2.5 vs.1.0; all P<0.001). Baseline demographic characteristics and laboratory values for both the non-event and event groups within the IM and OBGYN cohorts are presented in Supplementary Tables 2 and 3, respectively.
Model Performance
The VC-MAES tool consistently demonstrated superior performance over the NEWS and MEWS, as shown by both AUROC and AUPRC results across all evaluated time points. Six hours prior to the MAEs, the VC-MAES achieved an AUROC of 0.918 (95% CI, 0.912–0.924) and an AUPRC of 0.352 (95% CI, 0.330–0.374). In comparison, the NEWS yielded an AUROC of 0.797 (95% CI, 0.784–0.810) and an AUPRC of 0.124 (95% CI, 0.110–0.139), whereas the MEWS attained an AUROC of 0.722 (95% CI, 0.707–0.737) and an AUPRC of 0.079 (95% CI, 0.069–0.090; bootstrap test, P<0.001) (Figure 2). This trend remained consistent for the 12- and 24-hour intervals preceding the event. A comprehensive comparison of the results is presented in Table 2 and Figure 3.
For individual events, such as UIT, cardiac arrest, or death, the VC-MAES again outperformed both the NEWS and MEWS. For UIT, the VC-MAES achieved an AUROC of 0.890, compared with 0.698 for the NEWS and 0.627 for MEWS (bootstrap test, P<0.001). For prediction of death, the VC-MAES attained an AUROC of 0.994, outperforming both the NEWS (0.937) and the MEWS (0.850; bootstrap test, P<0.001). For cardiac arrest, the VC-MAES demonstrated an AUROC of 0.758 compared with 0.699 for the NEWS and 0.641 for the MEWS (bootstrap test, P<0.001).
In the IM subgroup (n=2,177), the VC-MAES outperformed the NEWS and MEWS, achieving an AUROC of 0.852 (95% CI, 0.841–0.862) and an AUPRC of 0.350 (95% CI, 0.327–0.375). In contrast, the NEWS produced an AUROC of 0.762 (95% CI, 0.748–0.775) and AUPRC of 0.119 (95% CI, 0.104–0.135), whereas the MEWS produced an AUROC of 0.697 (95% CI, 0.682–0.712) and AUPRC of 0.076 (95% CI, 0.067–0.086) (Figure 4A and B).
Similarly, in the OBGYN subgroup (n=3,862), the VC-MAES demonstrated the highest discriminative ability, with an AUROC of 0.964 (95% CI, 0.943–0.982) and an AUPRC of 0.388 (95% CI, 0.303–0.472). In contrast, the NEWS yielded an AUROC of 0.928 (95% CI, 0.891–0.961) and an AUPRC of 0.299 (95% CI, 0.225–0.385), whereas the MEWS attained an AUROC of 0.864 (95% CI, 0.816–0.912) and an AUPRC of 0.196 (95% CI, 0.139–0.267) (Figure 4B).
When excluding the 418 DNR patients, which resulted in 5,472 encounters, the VC-MAES model maintained superior performance with an AUROC of 0.862 (95% CI, 0.848–0.874) and an AUPRC of 0.032 (95% CI, 0.024–0.045). In comparison, the NEWS achieved an AUROC of 0.730 (95% CI, 0.702–0.758) and an AUPRC of 0.026 (95% CI, 0.018–0.036), while the MEWS demonstrated an AUROC of 0.700 (95% CI, 0.674–0.724) and an AUPRC of 0.020 (95% CI, 0.012–0.030).
Threshold-Based Performance and False-Alarm Reduction
Table 3 presents the performance metrics for the VC-MAES, NEWS, and MEWS across the key cutoff values. Compared with MEWS at a cutoff of 3 (specificity: 92.4%, sensitivity: 48.4%, PPV: 4.1%, NNE: 24.4), VC-MAES at a cutoff of 30 (specificity: 94.3%, sensitivity: 65.8%, PPV: 7.1%, NNE: 14.1) demonstrated improved specificity and sensitivity, with a 42% reduction in NNE. In terms of false-positives per true positive (FPpTP; calculated as NNE – 1), this improvement corresponds to a 44% reduction. Similarly, at a higher specificity range (approximately 97%), the MEWS at a cutoff of 4 (sensitivity: 36.5%, PPV: 7.3%, NNE: 13.7) was outperformed by the VC-MAES at a cutoff of 40 (sensitivity: 58.6%, PPV: 11.2%, NNE: 8.9), achieving a 35% reduction in NNE and a 38% reduction in FPpTP.
A similar trend was observed when comparing the VC-MAES and NEWS. At a specificity range of 96–98%, the NEWS at a cutoff of 5 (sensitivity: 45.2%, PPV: 7.8%, NNE: 12.8) was surpassed by the VC-MAES at a cutoff of 50 (sensitivity: 52.4%, PPV: 17.5%, NNE: 5.7), reflecting a 55% reduction in NNE and a 60% reduction in FPpTP. Even at the highest specificity level (≥99%), the NEWS at a cutoff of 7 (sensitivity: 28.5%, PPV: 16.3%, NNE: 6.1) was outperformed by the VC-MAES at a cutoff of 70 (sensitivity: 37.5%, PPV: 39.5%, NNE: 2.5). This represents a 59% reduction in NNE and a 71% reduction in FPpTP.
Associations between Baseline EWSs and MAEs or pLOS
Using the patient episode definition, we performed binary logistic regression with the first recorded (baseline) VC-MAES, NEWS, and MEWS to evaluate their associations with MAEs and pLOS. A pLOS was defined as a stay ≥7 days, corresponding to the 75th-percentile. Baseline VC-MAES showed a strong association with both MAEs and pLOS. Specifically, for MAEs, baseline VC-MAES had an odds ratio (OR) of 2.065 (95% CI, 1.871–2.280), whereas baseline NEWS and baseline MEWS had ORs of 1.051 (95% CI, 1.045–1.057) and 1.064 (95% CI, 1.054–1.073), respectively (Table 4).
After adjusting for age, sex, and body mass index, baseline VC-MAES continued to exhibit the strongest association with both MAEs and pLOS. For MAEs, baseline VC-MAES achieved an adjusted OR of 1.484 (95% CI, 1.333–1.651), surpassing those of the NEWS (adjusted OR, 1.032; 95% CI, 1.026–1.038) and MEWS (adjusted OR, 1.041; 95% CI, 1.032–1.052). Detailed comparisons of ORs and confidence intervals for pLOS are available in Supplementary Table 4.
DISCUSSION
In this prospective observational external validation study, we tested and validated the VC-MAES, a deep-learning-based EWS, in general ward patients, including a substantial proportion of OBGYN patients. Overall, the VC-MAES demonstrated superior predictive performance for MAEs, significantly outperforming both the NEWS and MEWS. Notably, this advantage was also pronounced in the OBGYN subgroup, highlighting its strong discriminative performance in a population seldom investigated in previous AI-based EWS studies. This finding underscores the potential applicability of the VC-MAES in both low- and high-risk patients admitted to the general wards, as demonstrated by its consistent performance in both the OBGYN and IM subgroups. Furthermore, the baseline VC-MAES at admission was a predictor of both MAEs and pLOS, suggesting its potential value as a prognostic and triage tool upon patient arrival.
External validation is essential to confirm the reliability and generalizability of AI models, as they frequently demonstrate diminished performance in external validation settings. This performance degradation often results from overfitting during training or shifts in the distribution of input features, a phenomenon referred to as “dataset shift” [24,25]. Recent external validation studies have demonstrated significant performance degradation in clinical deterioration prediction models [26,27], highlighting the challenges of maintaining model performance across settings and patient demographics. These findings emphasize the importance of rigorous external validation to identify potential biases and guarantee the robustness and generalizability of models across diverse clinical contexts.
The VC-MAES was originally trained using medical records from a tertiary academic medical center in the Republic of Korea. Although the current study was also conducted at a tertiary academic medical center in the Republic of Korea, this hospital was located in a province with distinct healthcare systems and patient demographics. Despite this substantial data shift, VC-MAES maintained strong performance, suggesting its potential for broader applicability across various clinical settings. In our study, the overall prevalence of MAEs was 3.6%, consistent with previously reported ranges of 3%–9% [28]. However, the incidences of MAEs differed significantly for the two subspecialties assessed: 0.26% for the OBGYN services and 9.5% for the IM services. Despite these differences in MAE incidence, the VC-MAES maintained strong performance, achieving an AUROC of 0.964 and an AUPRC of 0.388 in the OBGYN subgroup, respectively, and an AUROC of 0.852 and an AUPRC of 0.350 in the IM subgroup, demonstrating its applicability to both low- and high-risk patient populations. Notably, the VC-MAES achieved the best EWS in the OBGYN subgroup, mirroring findings from a recent multicenter study of an AI-based EWS by Churpek et al. [29], who reported a slightly higher performance among female patients than among male patients (AUROC, 0.844 vs. 0.824) and particularly strong results in obstetric encounters (AUROC, 0.909).
Previous studies have indicated that a significant barrier to healthcare provider adoption of AI-based algorithms is the concern about increased false alarms from continuous monitoring, which could lead to alarm fatigue and workflow disruption [30]. We found that the VC-MAES model could reduce false-positive MAE predictions by up to 71%, improving clinical efficiency by minimizing unnecessary evaluations. By curtailing superfluous alerts, healthcare facilities can optimize resource allocation, enabling providers to focus on critical patient care tasks.
This study has several limitations. First, as a non-interventional, single-center study, it was inherently susceptible to various biases and confounding variables. Nonetheless, the prospective design helped mitigate some biases commonly associated with retrospective validation studies, providing a useful foundation for future interventional research. Second, the validation datasets were derived from a single hospital in the Republic of Korea, which may limit the generalizability of our findings to other countries with different healthcare systems and patient demographics. Additionally, the study population was predominantly female, largely because of the substantial number of patients from the OBGYN department. This sex imbalance also limits the applicability of these results to other clinical settings. Third, although the VC‐MAES outperformed the NEWS and MEWS, the low incidence of cardiac events and overall adverse events in the OBGYN population limited further sub-analyses and more comprehensive verification of the model performance in that subgroup. Fourth, the model performance can be affected by the quality and completeness of EHR data, which vary across institutions. Although the VC‐MAES uses the LOCF method and normal‐value imputation for missing data, this approach may not fully capture the clinical context. For instance, in this study, most GCS scores were missing and were imputed to be normal (GCS=15) to maintain consistency in the analyses, which may have influenced the model’s overall performance. Finally, the study did not include an interventional component; therefore, we were unable to assess the impact of the VC-MAES on clinical workflow, patient outcomes, or physician engagement in a real-world environment.
In conclusion, our study demonstrated that the VC-MAES significantly outperformed traditional scoring systems in predicting MAEs in hospitalized patients. Despite challenges such as data shifts, the VC-MAES maintained strong performance across clinical settings, showing the potential to reduce false-positive predictions of MAEs and enhance patient outcomes. Future research should focus on validating these findings in diverse clinical settings to ensure the robustness and generalizability of the model. Interventional studies are required to assess the real-world impact of the VC-MAES on patient outcomes, clinical workflow, and physician engagement.
KEY MESSAGES
▪ In this prospective external validation study, VitalCare - Major Adverse Event Score (VC-MAES)—a deep-learning-based early warning system—demonstrated superior predictive accuracy for major adverse events compared with traditional early warning scores (National Early Warning Score and Modified Early Warning Score).
▪ Despite the markedly different demographic profiles of general ward patients in internal medicine and obstetrics and gynecology settings—representing high- and low-risk cohorts, respectively— VC-MAES demonstrated high predictive performance, underscoring its potential for broader generalizability.
▪ VC-MAES also reduced false-positives by up to 71%, suggesting the possibility of enhanced clinical efficiency; however, further studies are warranted to confirm its impact on workflow and patient outcomes.
XML Download