PDF
Citation
Print
Introduction
Due to population aging and chronicization of diseases, the severity of diseases among patients admitted to intensive care units (ICUs) has been increasing. Treatment costs have also been increasing for these patients. Since ICUs have limited medical resources, an objective patient triage protocol and evaluation of the severity of a condition must be developed to minimize unnecessary use of ICUs and allow for efficient use of the limited healthcare resources.
Prognostic scoring systems have been developed to estimate the in-hospital mortality of ICU patients [1-4]. For this reason, over the last three decades, severity scoring systems, such as the Acute Physiology and Chronic Health Evaluation (APACHE) score, the Simplified Acute Physiology Score (SAPS), and the Mortality Probability Model (MPM), have been attempted to be used as a critical care triage criterion beyond predicting hospital mortality in critically ill patients [5].
The APACHE scoring system developed by Knaus et al. [1] achieved higher calibration than Zimmerman et al. [6] after a series of improvements, suggesting that APACHE IV is more accurate [3,7]. According to recent studies, APACHE IV exhibits satisfactory discriminatory performance both in the United States, where it was first developed, and outside the United States [8-11]. The older APACHE II model has been validated in Korean populations; it exhibits poor calibration and modest discrimination for hospital mortality [12]. Whereas, the performance of the APACHE IV has not been sufficiently examined.
This study aimed to investigate the suitability of APACHE IV severity scores and MPMs in an ICU within a tertiary general hospital, by analyzing the relationships among APACHE IV scores at the time of the admission, the predicted mortality rate, and the actual mortality rate comparing with APACHE II model. This study also verified the usefulness of the APACHE IV score as a standard triage protocol for admission in the ICU.
Materials and Methods
1) Patients
The study was conducted in a 1,200-bed capacity referral hospital with four adults’ ICU: medical, surgical, emergency, and cardiac ICU, respectively. Two dedicated intensivists supervised all of 56 ICU beds in all ICUs on a semi-closed system. All patients were admitted between August 1, 2013 and July 31, 2014. The same type of patients suggested by Zimmerman et al. [6] for the development of APACHE IV were involved in this study. We excluded patients if they were younger than 17 years old or if the primary outcome of hospital mortality was uncertain. We also excluded patients with an ICU stay <48 hours, patients with burns, patients missing an APACHE IV score on day 1 in the ICU, patients who were admitted for simple postcardiovascular intervention monitoring, and patients who gave do-not-resuscitate orders. Three hundred and sixty-four patients were screened during the study period and 318 eligible patients were enrolled for analysis (Figure 1). This study was approved by the institutional review board of Chungnam National University Hospital (No. 2015-06-053), in accordance with the Declaration of Helsinki.
2) Differences between APACHE II and APACHE IV model
APACHE II score is calculated based on 12 physiologic criteria and estimates risk based on data available within the first 24 hours of an ICU stay [13,14]. APACHE IV was designed to assess the severity of illness as well as the prognosis in the ICU and has 17 physiological criteria, adding new variables such as mechanical ventilation, thrombolysis, impact of sedation on Glasgow Coma Scale, rescaled Glasgow Coma Scale, PaO2/ FiO2 ratio and disease-specific subgroups, to the existing APACHE III variables [6]. Disease-specific scoring systems have been developed for several important subgroups treated in the ICU since an APACHE III model [15].
3) Data collection
Two critical care fellows and one trained nurse prospectively collected and reviewed electronic medical records. The electronic medical records provided all of the data required to predict the mortality rate using APACHE II and APACHE IV. Predicted hospital mortalities were calculated using the equations of APACHE II and APACHE IV as follows: logit for APACHE II = –3.517 + (APACHE II) × 0.146. The APACHE IV score and predicted mortality rate calculation on a website (http://www.mecriticalcare.net/icu_scores/apacheIV.php) were used in the present study.
4) Identification of risk factors independent of the APACHE IV score and triage model development
Cox proportional hazards regression was conducted to examine associations with death after adjustment for the APACHE IV score. Hazard ratios (HRs) were used to quantify the relationship between risk factors and death. An ICU triage model, which predicts hospital mortality, was constructed by combining the APACHE IV score and the other risk factors identified above the Cox regression models.
5) Statistical analysis
Descriptive statistics are presented as medians and interquartile ranges, or as numbers with percentages. A univariate logistic regression analysis was performed to evaluate any associations between various risk factors and hospital mortality. Among the variables used in the model, the risk factors with P-values less than 0.05 were selected for multivariate logistic regression analysis.
Discrimination is defined as the power to distinguish between survivors and non-survivors, and this was evaluated by receiver operating characteristic (ROC) analysis [6]. Calibration was defined as agreement between individual probabilities and actual outcomes. It was assessed using the Hosmer-Lemeshow goodness-of-fit C statistic with P-values greater than 0.05 indicating good calibration [7]. The standardized mortality ratio (SMR) was the ratio between the observed and predicted number of deaths. To test for statistical significance, we calculated 95% confidence interval (CI) according to the method described by Hosmer and Lemeshow [6]. We estimated HRs in univariate Cox proportional hazards models with 95% CIs and level of statistical significance. To test the discrimination ability of different combinations of parameters like APACHE IV, Charlson Comorbidity index (CCI), and department, we used the area under the receiver operating characteristic curve (AUROC) by Delong method.
Statistical analysis was performed using SPSS version 21.0 (IBM Corp., Armonk, NY, USA). STATA version 12.0 (StataCorp., College Station, TX, USA) was used for C-statistics and the R and MKmisc version 3.2.2 was used for H-statistics.
Results
1) Baseline characteristics of the population
The baseline characteristics of the patients are shown in Table 1. The total number of patients was 318, 79 of whom were non-survivors, exhibiting a 24.8% mortality rate. Among the mortality factors, age, CCI scores, unplanned ICU admission, use of vasoactive agents, acute respiratory distress syndrome (ARDS), severe sepsis or septic shock, APACHE II scores, and APACHE IV scores exhibited statistically significant differences between the survivors and non-survivors (P < 0.001). The mean age of the non-survivors was 70.7 ± 12.6 years, which was higher than that of the survivors (63.1 ± 15.5 years). The mean CCI score of the non-survivors was 5.4 ± 3.2 points, exceeding that of the survivors (3.4 ± 2.5 points), suggesting that the non-survivors had more comorbidities than the survivors. No statistically significant differences were found for sex, body mass index, and ICU lengths of stay between the survivors and non-survivors.
2) Performance of the APACHE IV and APACHE II models in the prediction of hospital mortality
Both prognostic models showed reasonable discrimination and calibration (Table 2). The AUROC of the APACHE IV and APACHE II models were 0.759 and 0.752. ROC curves for the two scoring systems are shown in Figure 2. Hosmer-Lemeshow goodness-of-fit test generated P-values >0.05 for both APACHE II and APACHE IV, indicating that the models were comparable. APACHE IV exhibited the same mortality prediction rate as the observed mortality. The APACHE II model exhibited a lower SMR than the APACHE IV model (SMR APACHE IV, 1.000 [95% CI, 0.789 to 1.250]; APACHE II, 0.991 [95% CI, 0.788 to 1.248]).
3) Performance characteristics of different combination of parameters for predicting hospital mortality
The multivariate logistic regression model was used to determine independent risk factors for hospital mortality by using all variables with a P-value <0.05 in the univariate model. Among the variables used in the analysis, the APACHE IV score (OR, 1.023; P < 0.001; 95% CI, 1.012 to 1.034), the CCI (OR, 1.200; P < 0.001; 95% CI, 1.053 to 1.368), ARDS (OR, 13.187; P < 0.001; 95% CI, 3.941 to 44.122) and unplanned ICU admission (OR, 2.239; P = 0.015; 95% CI, 1.169 to 4.287) were associated independently with hospital mortality (Table 3).
With a cutoff score of 93, the APACHE IV score predicted hospital mortality with the highest sensitivity and specificity. The highest sensitivity and specificity were observed for CCI scores greater than 3. Table 4 shows that the HRs for hospital mortality. Medical admission, ARDS, CCI >3, and an APACHE IV >93 score were significant risk factors for hospital mortality.
The explanatory power of APACHE IV scores >93 in predicting the hospital mortality rate was 44.1%; the model explained 44.1% of the total variance. With regards to models in which medical admission, which is a risk factor for hospital mortality, CCI scores >3, and APACHE IV scores >93 were added, the APACHE IV scores >93 and medical admission model had an explanatory power of 48.9%, and the APACHE IV scores >93, medical admission, and CCL >3 model had an explanatory power of 53.8%. Therefore, including risk factors in the models improved their explanatory power. However, the discriminative ability of the prediction models was not satisfactory to use them in a triage protocol for admission in the ICU, since all three models had a C index lower than 0.7 (Table 5).
Discussion
The present study aimed to investigate the suitability of APACHE IV severity scores and MPMs in the ICU of a tertiary general hospital. The study also evaluated the usefulness of APACHE IV scores as a single criterion of a triage protocol for admission in ICUs.
There has been much debate on the need for objective directives to follow for an ICU admission triage, which aims to efficiently provide critically ill patients with resources within ICUs [16-18]. A substantial amount of research has also been conducted on the efficacy of existing physiological scores in predicting mortality in ICUs [19-22]. Previous studies have proposed various triage protocols for ICU admission [23].
Since the APACHE scoring system is based on objective physiological factors, it eliminates the possibility of errors made by the user. It also allows for simultaneous comparison and prospective analyses of patients from different ICUs, and can be applied to a wide diversity of patients. Therefore, it was our hypothesis that the APACHE scoring system would be useful as the admission triage criterion.
This study showed that APACHE IV had good calibration and modest discrimination among the critically ill patients in a single center. Moreover, APACHE IV showed an SMR close to the actual mortality rate. Therefore, the suitability of APACHE IV for evaluating the severity of patients’ conditions and predicting their prognoses was verified in this urban referral hospital. It was also found that APACHE IV makes more accurate predictions of patients’ prognoses compared to APACHE II scores in even single center. This could be explained due to the advance of APACHE IV model using additional factors such as mechanical ventilation support, disease specific subgroup analysis, and the specific reason for ICU admission. Daley et al. [24] pointed out that APACHE II has been widely used for measuring ICU performance but this scoring system is not disease specific [25-29]. The same as that APACHE IV scoring system show more reliable prediction in Asia population were observed in other validation studies [30-33].
In aspect of the discriminative ability of new prediction models, the result exhibited unsatisfactory discrimination to use them in a triage protocol for admission in the ICU, as showing all three models had a C index lower than 0.7. There are some reasons why the new prediction model showed unsatisfactory discrimination as an ICU triage protocol. First, the semi-closed system of the ICUs in which the present study was conducted would have prevented lots of patients with a too-sick-to-benefit status from admitting to ICUs in advance by intensivists [34]. Second, the sample size was smaller than that of previous studies on triage models leading to insufficient statistical analysis [35]. Finally, despite establishing 93 as our cutoff score for APACHE IV, the higher score of a prognostic model has greater explanatory power. Though, the present study was limited to comparing the scores divided into only two ranges (<93 or ≥93).
This study had some strengths. The validity of APACHE IV in surgical and medical ICUs, emergency ICUs, and cardiovascular ICUs was evaluated by using data of the same type of patients suggested by Zimmerman et al. [6] for the development of APACHE IV. Second, the present study is meaningful in that it attempted to verify the validity of a standard prognostic scoring system for few domestic studies. Moreover, the results suggest that the APACHE IV system makes more accurate predictions of patients’ prognoses compared to APACHE II scores even in single centers. Finally, within the scope what we know, there is rare research demonstrating that the APACHE IV model would not be advisable as a single criterion for admission in Korea ICU.
However, the study results may contain selection bias since this study took place in only one institution. We could not obtain information on the survival status of patients who could not be monitored due to loss to follow up. And this study had a smaller sample size of 318 patients compared to the sample sizes used in previous studies [19,20,36].
According to our results, in order to establish objective criteria for admission to ICUs, overall clinical judgment of internal medicine patients, severity of comorbidities, main diagnosis type (ARDS, septic shock), age, likelihood of recovery, opinions of medical professionals, patients’ conditions, and prognoses may be helpful. And multilateral research involving larger patient populations and disease groups is essential.
In conclusion, in this work, the APACHE IV scoring system exhibits satisfactory discrimination and excellent calibration, but the result supposed that it was not appropriate to be used as a single criterion for ICU admission. Further research on determining the ICU admission priority of critically ill patients would be necessary.
XML Download