Evaluating the quality of medical care requires evaluating the effectiveness of the treatment provided to a patient. Hence, a valid evaluation must be preceded by assessments of the patient’s condition before the treatment is given [1]. In order to evaluate patient condition, intensive care units (ICUs) utilize severity scoring systems [2]. The quality of medical treatments in the ICU can be evaluated objectively by comparing actual mortality with predicted mortality using methods that take into account patient severity.

Severity scoring systems can be divided into two categories, depending on the specific system used to collect data. The first category includes the Acute Physiology and Chronic Health Evaluation (APACHE), Simplified Acute Physiology Score (SAPS), and Mortality Probability Model (MPM), all of which quantify the initial condition of the patient within 24 hours of ICU admission. The second category includes organ system failure (OSF), sequential organ failure assessment (SOFA), organ dysfunction and infection system (ODIN), and multiple organ dysfunction score (MODS), all of which measure patient condition repeatedly throughout the admission period.

Among these systems, the severity scoring systems most commonly used in the ICU are APACHE, SAPS, and MPM, which evaluate the initial condition [2]. While the measurement variables used by these systems were initially selected subjectively, all have selected statistically meaningful variables since 1990, thereby improving their performance [3].

Our institution established a severity correction model using APACHE II, which was updated in 1985, and is used to constantly monitor predicted mortality in the ICU via APACHE II scores. However, the perceived need for a new system to measure patient severity has increased over time. In 2014, the performance of APACHE IV and SAPS 3 was compared in a surgical ICU context [4].

In this study, we analyzed APACHE IV, SAPS 3, and MPM₀ III in order to determine which system best implements data about the severity of medical ICU patients. We utilized the selected system to create a severity correction model that can measure predicted mortality rates.

This research, approved by Institutional Review Board of Seoul National University Hospital (IRB No. H-1605-116-763), was a retrospective investigation that analyzed the discrimination and calibration of APACHE II, APACHE IV, SAPS 3, and MPM₀ III, when used for medical ICU patients.

The capacity of a severity scoring system to predict death varies between medical and surgical patients [5]. Such disparity presumably arises from the fact that the initial development of conventional systems such as APACHE and SAPS was targeted at large groups of patients with a variety of diseases. When these systems are implemented only for patients with a particular disease, additional factors other than the disease itself can affect predictions of death [6]. Therefore, in order to predict death rates more accurately, a scoring model that takes into account all of the different characteristics of patients in each ICU has been suggested [7-9].

In this study, we aimed to identify the severity scoring system that best reflects the characteristics of critically ill patients in medical ICUs. Two aspects of such systems were considered: discrimination and calibration. Discrimination in this context is the ability of the system to distinguish whether the subject has deceased or survived, represented by c-statistics. The systems with c-statistics closest to 1 are in descending order as follows: APACHE IV (AUC, 0.745), APACHE II (AUC, 0.729), SAPS 3 (AUC, 0.700), and MPM₀ III (AUC, 0.670). In another study focused on patients treated in our hospital’s surgical ICU, we found that the c-statistics of APACHE IV were lower than those of SAPS 3. Nonetheless, the value of APACHE IV was still 0.80, indicating sufficient discrimination capacity [4]. Another previous study also suggested superior discrimination of APACHE and SAPS compared to MPM, which may be due to the larger number of variables considered by APACHE and SAPS, thus allowing these systems to better calculate the intricate relationships between different factors measured in patients [2].

Calibration refers to the closeness between the predicted morality rate and the simple observed mortality rate, a probabilistic criterion used to evaluate the performance of each system when targeting all patients. In this study, all severity scoring systems satisfied P>0.05 according to the Hosmer-Lemeshow test, therefore signifying excellent performance. Given that P is closer to 1 when the predicted mortality is closer to the simple observed mortality, APACHE IV had the best performance, followed by SAPS 3, MPM₀ III, and APACHE II. In most domestic and international studies, all three systems demonstrated adequate calibration for predicting simple mortality [10]. Some studies, however, suggested poorer calibration, both in surgical patients [4] and internal medicine patients [2]. Groeger et al. [11] suggested that these conflicting results can be explained by differences in the study population.

In general, APACHE IV and SAPS 3 reflected the severity of patients better than MPM₀ III. This result may be due to the number of variables each system implements. If the system requires more variables, it takes a longer time for evaluators to input the values, and the time required by each evaluator varies significantly. If a system requires fewer variables, there are fewer discrepancies between different evaluators, and severity measurements can be completed in a relatively short time. These reasons may explain the benefits of using MPM₀ III. However, patient factors are currently automatically extracted from electronic medical records. This development eliminates the time-related disadvantages of systems that consider more variables, thereby making the ability to take full account of patient condition the most important criterion for choosing an appropriate severity scoring system. Thus, in this study we suggest a severity correction model (P<0.01) using APACHE IV.

The limitations of this study lie with the data collection system. Whereas many studies evaluating the performance of different severity scoring systems collected data using a prospective cohort system [12-14], we collected data retrospectively in this study. We initially aimed to analyze records from past patients in order to immediately implement our study results in ongoing clinics, but it took a long time to analyze the medical records for all scoring systems. Thankfully, the average severity measured by APACHE II of medical ICU patients was 24.3 in 2014, 23.1 in 2015, and 23.7 in 2016, so there was no significant difference.

ICU patient severity plays the most important role in determining whether the patient will survive or not, and that severity is significantly affected by underlying disease. However, when the underlying disease is the same between patients, the impacts of other major factors on severity decrease, which may lead to erroneous death predictions [11]. The significance of the present study lies in this idea. Instead of implementing conventional severity scoring systems, we established a death prediction model specifically for patients admitted to the ICU for the same reason. Therefore, ICU entry should predict patient death much more accurately. We further recommend that other ICUs in our hospital, including the cardiopulmonary ICU and emergency ICU, should implement a specific severity scoring system that fully takes into account the characteristics of patients unique to each unit.

Besides patient factors, other structural factors play major roles in determining mortality rates, including standardization guidelines for the treatment and an abundance of medical personnel such as doctors and nurses working in the hospital [15-17]. Therefore, the development of a predictive model for death will depend on underlying disease. Such predictive models will serve as capstones for evaluating factors that affect the quality of medical care.