PDF
Citation
Print
INTRODUCTION
In 2020, the disease burden attributed to injuries comprised 9.2% of Korea’s total disability-adjusted life years (DALYs).1 According to the Global Burden of Disease study, although Korea’s injury burden has decreased since 1990 (from 2,045 DALYs in 1990 to 1,550 DALYs in 2019), the proportion of injuries in the total burden of diseases is still higher than the global burden in 2019 (Korea 13.3% vs. Global 9.8%).2 In addition, injury is a critical health problem due to its bigger proportion of the burden attributed to premature death compared to the total disease burden (74.3% vs. 66.1%).2 Since implementing a national trauma care system in 2012, Korea has made significant efforts, including establishing regional trauma centers and the Korean Trauma Data Bank (KTDB).
Ensuring a reliable assessment of injuries is essential for effectively operating the trauma care system. Various trauma mortality prediction models and severity assessment methods, such as the International Classification of Diseases-based Injury Severity Score (ICISS), are employed. The Trauma and Injury Severity Score (TRISS), which incorporates demographics, Revised Trauma Score (RTS), and Injury Severity Score (ISS), is widely used as a mortality prediction model in trauma patients.3 However, the TRISS is labor-intensive due to the requirement for an Abbreviated Injury Scale (AIS) obtained by a trained injury coder.4 On the other hand, Extended ICISS, another commonly used model, is less labor-intensive as it incorporates demographics, RTS, and ICISS based on diagnosis codes and survival risk ratios (SRRs). In the Korean Trauma and Emergency System, the mortality based on the TRISS model is only available from 17 regional trauma centers that adopted the KTDB system. In addition, several trauma centers in Seoul, designated as final treatment centers for severe trauma patients in 2021, are not subjected to the KTDB system. Therefore, the extended ICISS model is a realistic option for a comprehensive assessment of all trauma injury patients across the country.
In a previous validity study, the Extended ICISS model showed non-inferior performance compared to the TRISS model in a blunt injury population.5 SRRs were derived using one-year discharge abstracts from 35 Korean emergency centers in 1996.5 Nakahara and Yokota6 suggested using country-specific SRRs and periodically updating them to reflect discrepancies across countries and over time. On the other hand, international collaborative efforts have been made to achieve pooled SRRs from seven countries (Australia, Argentina, Austria, Canada, Denmark, New Zealand, and Sweden).78 A comparison between the extended ICISS models in the Korean context, depending on the methods used to derive SRRs, would be worthwhile. Furthermore, the National Emergency Department Information System (NEDIS), a representative data collection system covering all emergency cases, lacks the inclusion of the Glasgow Coma Scale (GCS) necessary for calculating RTS. Instead, the NEDIS provides only the Alert, Voice, Pain, Unresponsive (AVPU) scale. Given the practical use of extended ICISS in Korea, model evaluations should consider replacing the GCS with the AVPU scale.
This study primarily aimed to compare the performance of the Extended ICISS models, incorporating updated domestic SRRs, with TRISS for comprehensive evaluation of trauma care outcomes. Additionally, we examined the effect of different methods for SRR calculation and explored the alternative use of the AVPU scale instead of the GCS to derive the RTS.
METHODS
Data source and study cohort
This study uses two types of datasets: 1) NEDIS to derive representative SRRs and 2) KTDB to estimate mortality prediction models. The target study cohort included trauma patients with S or T diagnosis codes based on the Korean version of the International Classification of Diseases, Tenth Revision between 2017 and 2021 (N = 188,590; Fig. 1). The exclusion criteria were as follows: 1) patients with no trauma injury diagnosis codes beginning with S or T (n = 2,313); 2) patients with only diagnosis codes such as foreign body (T15–T16), intoxication (T36–T659), unspecified injuries (T66–T789), or trauma complications (T793, T798, T799, T80–T983) (n = 281); 3) patients transferred to other institutions (n = 64,066); 4) voluntarily discharged (n = 2,235); 5) patients known as unidentified status, including discharge without permission (n = 273); 6) patients with missing information about their status at the time of discharge from the emergency room (n = 156). Additionally, we excluded cases with missing model parameters for complete case analysis (CCA) (n = 14,410). The final target population was 104,856. We also explored model performance in subgroup cohorts, including all trauma patients aged ≥ 18, blunt trauma patients aged ≥ 18, penetrating trauma patients aged ≥ 18, and burn and other trauma patients aged ≥ 18.
Training and validation cohorts
Using stratified random sampling, we divided the target cohort dataset into training and validation datasets in an 8:2 ratio: training (n = 83,889) and validation (n = 20,967). Strata sampling considered the following characteristics: adult (≥ 18 years), sex, and injury mechanism (blunt, penetrating, and burn and other trauma).
Model parameters
We compared six major trauma mortality prediction models using demographics and trauma scoring systems as parameters. Every model included demographics (age and sex) and RTS, a physiologic-based triage score. First, we assessed the validity of the extended ICISS adopting updated SRRs by comparing them with the TRISS (TRISS vs. extended ICISS 1–5). Second, we compared the model's performance benefits using international SRRs (ICISSInt-4dight, extended ICISS 1) and domestic SRRs (ICISSKor-4digit, extended ICISS 2). Third, our models considered applying truncated diagnosis codes at 4-digit (ICISSKor-4digit) and full codes (i.e., no truncated code, ICISSKor-full) regarding model performance (extended ICISS 2 vs. extended ICISS 3; extended ICISS 4 vs. extended ICISS 5). Fourth, we assessed the utilization of the AVPU scale instead of the GCS to derive the RTS (extended ICISS 2 vs. extended ICISS 4; extended ICISS 3 vs. extended ICISS 5). While the modified RTS was recalculated using matched values between the AVPU and GCS scores, we used the original RTS from the KTDB.9 Lastly, we explored the additional ICISS model by including a body region of injury, specifically head injuries, obtained from the AIS codes.
SRR and the ICISS
The models utilized three types of SRRs. We used pooled SRRs based on 4-digit diagnosis codes from hospitalization datasets in seven countries (Australia, Argentina, Austria, Canada, Denmark, New Zealand, and Sweden).78 Korean SRRs were calculated from the NEDIS dataset between 2017 and 2021 depending on the type of diagnosis codes (truncated at 4-digit and full codes), referencing the methods described in previous research.57 The method used to calculate the ICISS has been presented in a previous study.10
Statistical analysis
General characteristics between the groups were compared using independent t-tests and χ2 tests for continuous and categorical variables, respectively. Predictive mortality was analyzed using logistic regression modeling. We fit the model to the validation datasets using the coefficients obtained from the training datasets, depending on the cohort. Model performance was evaluated from the perspectives of calibration and overall performance. Discriminative ability was measured using the area under the receiver operating characteristic curve (AUC) calculated from the receiver operating characteristic curve, along with accuracy, sensitivity, and specificity.11 For overall performance, we calculated the Brier score, which is the mean squared error of the estimators.12 All statistical analyses were performed using SAS 9.4 (SAS Institute, Cary, NC, USA). P values < 0.05 were considered statistically significant.
RESULTS
Characteristics of the study cohort
There were no statistically significant differences in the general characteristics between the training and validation cohorts (Table 1). Within the training cohort of all trauma patients, we observed significant differences in demographics, injury mechanism, trauma severity, and service features (length of stay and intensive care unit [ICU] admission) between survived and deceased patients (Table 2). Non-survivors after trauma had a higher mean age (survived vs. deceased, mean [standard deviation {SD}], 53.1 [21.4] vs. 66.7 [17.6], P < 0.001) and ISS (7.8 [6.7] vs. 24.4 [12.2], P < 0.001). They also showed lower values in the original RTS (7.8 [0.4] vs. 6.0 [1.8], P < 0.001), modified RTS (7.7 [0.4] vs. 6.0 [1.7], P < 0.001), ICISSInt-4digit (0.934 [0.093] vs. 0.754 [0.166], P < 0.001), ICISSKor-4digit (0.907 [0.139] vs. 0.663 [0.238], P < 0.001), ICISSKor-full (0.904 [0.145] vs. 0.644 [0.249], P < 0.001), and length of stay (16.0 [18.0] vs. 14.6 [26.3], P < 0.001) than survivors. The proportions of males (51,671 [64.7%] vs. 2,780 [69.6%], P < 0.001) and ICU admissions (17,030 [21.3%] vs. 3,074 [76.9%], P < 0.001) were higher among deceased trauma patients. The distribution of injury mechanisms and severity based on the ISS also showed a significant difference (P < 0.001).
Table 1
Descriptive statistics of training and validation cohorts
Table 2
Comparison of general characteristics between survived and deceased population in the training cohort
Model performance
In all trauma patients, the TRISS model showed the highest AUC estimates in both the training (AUC, 0.949; 95% confidence interval [CI], 0.945–0.952) and validation cohorts (0.950; 95% CI, 0.943–0.957) (Fig. 2, Table 3). In the training cohort, the extended ICISS model 3 (including the original RTS with GCS scores and ICISS based on the Korean full diagnosis code) presented the second-highest AUC value (0.938; 95% CI, 0.934–0.942). On the other hand, in the validation dataset, extended ICISS model 1, using international SRRs, ranked second (AUC, 0.939; 95% CI, 0.932–0.947). There was no statistical difference in AUC values between the extended ICISS models, considering the overlapping CIs, in both the training and validation datasets.
Fig. 2
Comparison area under the curve estimates of models in training datasets.
TRISS = trauma score and injury severity score, ICISS = International Classification of Diseases-based injury severity score.
Table 3
Summary of model performance
TRISS, age + sex + original RTS + ISS; Extended ICISS 1, age + sex + original RTS + ICISSInt-4dight; Extended ICISS 2, age + sex + original RTS + ICISSKor-4digit; Extended ICISS 3, age + sex + original RTS + ICISSKor-full; Extended ICISS 4, age + sex + modified RTS + ICISSKor-4digit; Extended ICISS 5, age + sex + modified RTS + ICISSKor-full.
AUC = area under the receiver operating characteristic curve, CI = confidence interval, TRISS = trauma score and injury severity score, ICISS = International Classification of Diseases-based injury severity score, RTS = revised trauma score.
Among other model performance measures, TRISS using ISS outperformed the extended ICISS models. In the training cohort, TRISS had an accuracy of 97.1%, while the other models ranged from 96.5% to 96.8% (Table 3). The sensitivity of TRISS was 51.0%, compared to 42.2–46.9% for the extended ICISS models. The TRISS model exhibited higher specificity than the other models, with a difference of approximately 0.1% in both the training and validation datasets. The overall performance of the TRISS model was also slightly better than that of the other extended ICISS models in training (Brier score, 0.024 vs. 0.026–0.027) and validation datasets (0.024 vs. 0.027–0.028).
While Korean domestic SRRs showed higher performance than international SRRs in the training cohort (extended ICISS 1 vs. extended ICISS 2, AUC, 0.936; 95% CI, 0.932–0.940 vs. 0.937; 95% CI, 0.933–0.941), the trend was reversed in the validation cohort (0.939; 95% CI, 0.932–0.947 vs. 0.938; 95% CI, 0.932–0.947) (Fig. 2, Table 3). Models using full diagnosis codes (extended ICISS 3 and extended ICISS 5; AUC, 0.938; 95% CI, 0.934–0.942) slightly outperformed those using truncated codes (extended ICISS 2 and extended ICISS 4, AUC, 0.937; 95% CI, 0.933–0.941) in the training cohort. The validation dataset showed a similar trend, with insignificant differences (0.935–0.939, 95% CI, 0.928–0.946 vs. 0.935–0.938, 95% CI, 0.927–0.947). When replacing the GCS with the AVPU scale in the calculation of RTS, no meaningful performance loss was observed in the training cohort (extended ICISS 2 and extended ICISS 3, 0.937–0.938, 95% CI, 0.933–0.942 vs. extended ICISS 4 and extended 5, 0.937–0.938, 95% CI, 0.933–0.942). In the validation dataset, the AUC values dropped slightly in the model using the modified RTS (extended ICISS 2 and extended ICISS 3, 0.938–0.939, 95% CI, 0.931–0.947) compared with the models using the original RTS (extended ICISS 4 and extended ICISS 5, 0.935, 95% CI, 0.927–0.943). All the parameter coefficient estimates of the individual models are presented in Supplementary Table 1.
Subgroup analysis
As seen in the all-trauma cohort, the TRISS model outperformed in subgroup cohorts: the adult population (AUC, 0.946; 95% CI, 0.942–0.949; Brier score, 0.025) and the adult blunt trauma population (AUC, 0.943; 95% CI, 0.939–0.947; Brier score, 0.027; Fig. 2, Table 4). We also cannot observe the ICISS model’s performance improvement when head injuries are included as a parameter in both adult blunt training (without head injuries AUC, 0.930–0.932 vs. with head injuries AUC, 0.929–0.931) and validation cohorts (without head injuries AUC, 0.933–0.934 vs. with head injuries AUC, 0.932–0.934) (Table 4, Supplementary Table 2). Although the penetrating and burn sub-trauma cohorts showed different trends regarding the superiority of the TRISS model, the CIs overlapped, representing insignificant differences. In the penetrating trauma training cohort, TRISS had an AUC of 0.922 (95% CI, 0.873–0.972), while the other models ranged from 0.916 to 0.937 (95% CI, 0.862–0.980). A trivial difference was observed in the overall performance between TRISS and the other models in both the training (Brier score, 0.005 vs. 0.006) and validation cohorts (0.006 vs. 0.006–0.007) for adult penetrating trauma patients.
Table 4
Summary of model performance in subgroup cohorts
TRISS, age + sex + original RTS + ISS; Extended ICISS 1, age + sex + original RTS + ICISSInt-4dight; Extended ICISS 2, age + sex + original RTS + ICISSKor-4digit; Extended ICISS 3, age + sex + original RTS + ICISSKor-full; Extended ICISS 4, age + sex + modified RTS + ICISSKor-4digit; Extended ICISS 5, age + sex + modified RTS + ICISSKor-full.
AUC = area under the receiver operating characteristic curve, CI = confidence interval, TRISS = trauma score and injury severity score, ICISS = International Classification of Diseases-based injury severity score, RTS = revised trauma score.
The use of domestic SRRs showed slightly higher AUC estimates than the international SRRs in the individual training data subgroup analysis, although the difference was not significant (Fig. 2). For example, the AUC of the extended ICISS 1 was 0.932 (95% CI, 0.928–0.936) in all trauma patients aged ≥ 18 years, whereas the estimate was 0.933 (95% CI, 0.929–0.937) in the extended ICISS 2 (Table 4). There was also no significant difference in the AUC estimates between the models using domestic and international SRRs in the validation datasets. In the validation dataset of the all-trauma adult population, discriminative performance was the same (0.935; 95% CI, 0.927–0.943) between the extended ICISS 1 and 2. However, international SRRs showed marginally better overall performance than domestic SRRs in the training dataset (extended ICISS 1 vs. extended ICISS 2, Brier score, 0.027 vs. 0.028). The Brier scores were identical in the validation dataset for the all-trauma adult patients.
Although the models using full diagnosis codes for the ICISS (extended ICISS 3 and 5) demonstrated slightly higher AUC estimates than those using truncated codes (extended ICISS 2 and 4) in all subgroup training datasets, there was no significant difference considering overlapping CIs in all subgroup analyses (Fig. 2). Additionally, this difference was inconsistent or insignificant in the validation dataset (Table 4). We could not find any robust differences in the Brier score between the models using the full and truncated codes in all subgroups. Using the AVPU scale (extended ICISS 4 and 5) instead of the GCS scores (extended ICISS 2 and 3) showed an insignificant difference in all subgroup training and validation datasets. For instance, the model using the AVPU scale and GCS scores had the same level of performance (extended ICISS 2 vs. extended ICISS 4, AUC, 0.933; 95% CI, 0.929–0.937) in the training cohort of the all-trauma adult patients. In the validation dataset, the extended ICISS 3, including the GCS scale (AUC, 0.935; 95% CI, 0.927–0.943), showed a higher estimate than the extended ICISS 5 using the AVPU scale (AUC, 0.934; 95% CI, 0.926–0.941), although the difference was insignificant despite overlapping CIs. Overall, the models using the GCS scores were better than those using the AVPU scale, as indicated by the lower Brier scores.
DISCUSSION
This study provided updated information on the validity of the extended ICISS compared with the commonly used TRISS model, considering practical points related to SRRs and RTS. First, the overall discriminative performance of all models can be considered outstanding (AUC ≥ 0.9), except in the validation cohort of penetrating trauma patients aged ≥ 18 years (AUC, 0.879–0.914).13 According to the systematic review of mortality prediction models in trauma patients, the AUC estimates ranged from 0.712 to 0.998 in the TRISS models and between 0.939 and 0.977 in extended ICISS models.3 Therefore, our models exhibit comparable and non-inferior discriminative performance. Second, we reaffirmed the feasibility of the extended ICISS models, despite their slightly lower performance compared to the TRISS model in both training and validation cohorts (training cohort, AUC, 0.936–0.938 vs. 0.949; validation cohort, AUC, 0.935–0.939 vs. 0.950). Third, when comparing models based on the methods used to derive SRRs, only marginal differences were found between them. Although Korea-specific SRRs yielded higher AUC estimates than international SRRs, no significant difference was observed between them in the training data (extended ICISS 1 vs. extended ICISS 2; Fig. 2). Using full diagnosis codes to derive the SRRs showed a similar trend. Models using full diagnosis codes showed a small performance improvement but were not statistically significant in the training data (extended ICISS 2 vs. extended ICISS 3 or extended ICISS 4 vs. extended ICISS 5; Fig. 2). Lastly, the alternative use of the AVPU scale instead of the GCS in calculating the RTS did not degrade model performance (extended ICISS 2 and extended ICISS 3, 0.937–0.938, 95% CI, 0.933–0.942 vs. extended ICISS 4 and extended 5, 0.937–0.938, 95% CI, 0.933–0.942), as presented in Table 3. Consequently, we confirmed the possibility of using the extended ICISS model and the NEDIS data to evaluate trauma care outcomes comprehensively.
This study holds significance from both academic and practical perspectives. We explored SRR calculation methods (i.e., international SRRs and truncated diagnosis codes), which have rarely been discussed in the Korean trauma care system. Despite a minor decrease in model performance using pooled SRRs, the use of international SRRs was supported owing to the advantage of comparability between countries.78 For example, a study in Australia used truncated codes.14 But, our study showed the benefits of using domestic SRRs based on full codes over international SRRs based on truncated codes. Furthermore, we explored the alternative use of the AVPU scale, a more applicable tool for assessing consciousness levels, for the extended ICISS model.1516 Considering the insignificant difference between models incorporating the GCS and AVPU scale, utilizing the modified RTS with AVPU is feasible within the NEDIS dataset. Regarding demographics as a component of the prediction model, we included sex, reflecting a lower mortality rate in females with severe trauma.17
Our study has some limitations and challenges. First, our main target population was all trauma patients, regardless of the mechanism of trauma or age, considering the comprehensive use of the prediction model in the overall population. Depending on the purpose, specific populations can be targeted, such as blunt trauma patients or severe trauma cases. Otherwise, a study could explore a model with the trauma mechanism as an explanatory factor, similar to the previous TRISS model.3 Second, we only considered the in-hospital mortality rate as the outcome of interest in this research. Future studies can investigate various models depending on outcomes (e.g., inpatient or 30-day outcomes).18 Third, several studies have evaluated the inclusion of a comorbidity tool, although we did not consider this in our models. Gabbe et al.19 found no significant improvement by including the Charlson Comorbidity Index (CCI) in an in-hospital mortality prediction model using TRISS. On the other hand, the CCI contributed to performance improvement in the model predicting 30-day mortality.20 Therefore, a study comparing the benefits of including comorbidities in a trauma mortality prediction model would be valuable. Fourth, we investigated the potential improvement of the extended ICISS by adding head injury as a new parameter, given the scale’s limitation of not considering the body of injuries.21 However, we could not observe an improvement in the model’s performance. Although current NEDIS does not include information on the body region of injury, we analyzed a preliminary model utilizing AIS codes. Fifth, this study does not include a calibration test due to the limitations of the Hosmer-Lemeshow test in a large sample.1122 The power of the Hosmer-Lemeshow test tended to increase with larger sample sizes.1123 Introducing a novel statistical method (e.g., the method suggested by Nattino et al.23) would be a feasible alternative for large datasets. Lastly, we adopted CCA after examining the missing data patterns. Although the dropout rate was 12.1%, no specific missing patterns were observed. Considering 5% of the threshold for acceptable missing data as a rule of thumb, our assumption of CCA (i.e., missing completely at random or missing at random) may be exaggerated.2425 According to the systematic review of missing data in trauma registries, 78.8% of publications reported CCA.26 Nevertheless, a previous study reported more severe injuries among patients with missing GCS scores.20 Therefore, we recommend introducing various techniques such as multiple imputations or best and worst-case scenarios to address problems attributed to missing values.2728
This study provides evidence that the extended ICISS is a usable model to predict trauma mortality based on overall and discriminatory performances. In particular, we demonstrated the feasibility of the extended ICISS in the NEDIS by exploring simple and practical models. Our practical ICISS model can be used to assess and evaluate the comprehensive Korean trauma system using mortality predictions and severity classifications.
XML Download