Does the Probability of Survival Calculated by the Trauma and Injury Severity Score Method Accurately Reflect the Severity of Neurotrauma Patients Admitted to Regional Trauma Centers in Korea?

Mahnjeong Ha; Seunghan Yu; Jung Hwan Lee; Byung Chul Kim; Hyuk Jin Choi

doi:10.3346/jkms.2023.38.e265

Abstract

Background

Assessing and improving the quality of trauma care is crucial in modern trauma systems and centers. In Korea, evaluations of regional trauma centers are conducted annually to assess and improve trauma management quality. This includes using the Trauma and Injury Severity Score (TRISS) method to calculate the W-score and mortality Observed-to-Expected ratio (O:E ratio), which are used to evaluate the quality of care. We analyzed the potential for overestimation of the probability of survival using TRISS method for patients with neurotrauma, as well as the potential for errors when evaluating and comparing regional trauma centers.

Methods

We included patients who visited the regional trauma center between 2019 and 2021 and compared their probability of survival of the TRISS method, W-score, mortality O:E ratio, and misclassification rates. The patient groups were further subdivided into smaller subgroups based on age, Glasgow Coma Scale (GCS), and Injury Severity Score, and comparisons were made between the neurotrauma and non-neurotrauma groups within each subgroup.

Results

A total of 4,045 patients were enrolled in the study, with 1,639 of them having neurotrauma. The neurotrauma patient group had a W-score of −0.68 and a mortality O:E ratio of 1.044. The misclassification rate was found to be 13.3%, and patients with a GCS of 8 or less had a higher misclassification rate of 37.4%.

Conclusion

The limitations of using the TRISS method for predicting outcomes in patients with severe neurotrauma are exposed in this study. The TRISS methodology demonstrated a high misclassification rate of approximately 40% in subgroups of patients with GCS less than 9, indicating that it may be less reliable in predicting outcomes for severely injured patients with low GCS. Clinicians and researchers should be cautious when using the TRISS method and consider alternative methods to evaluate patient outcomes and compare the quality of care provided by different trauma centers.

INTRODUCTION

Trauma quality assessment and improvement are critical components of any modern trauma system or trauma center. Trauma patients often require complex and time-sensitive care, which means that the quality of care provided can have a significant impact on patient outcomes. The national concern for trauma patients was relatively inconsequential, and the database of trauma patients was not established compared to developed countries. The preventable death rate for trauma patients in Korea was significantly higher, exceeding 35%, than that of developed countries.1 Since the 2010s, the South Korean Ministry of Health and Welfare has been interested in improving the management of trauma patients and started the regional trauma center (RTC). Currently, 17 RTCs have been designated, and 15 have been opened. After the establishment of RTCs, the Ministry of Health and Welfare has been conducting annual evaluations of RTCs to assess and improve the management of trauma. One of the evaluation criteria is the assessment of the quality of care, and the W-score and mortality Observed-to-Expected ratio (O:E ratio) calculated using the Trauma and Injury Severity Score (TRISS) method are used for this purpose. According to the TRISS method, the probability of survival (Ps) is calculated using the patient’s age, physiological parameters, and anatomical injury severity.

Through the mortality case panel discussions, our institution recognized the low correlation between expert opinions and Ps for patients with neurotrauma. We realized that the severity of patient’s neurotrauma was not properly reflected. Furthermore, the assessment of quality management for the RTCs involves the use of W-scores and mortality O:E ratio, which are based on TRISS methodology. Therefore, it was determined that errors may arise when evaluating and comparing RTCs using TRISS method. We conducted a study to analyze this mismatching.

METHODS

Database

The Korea Trauma Data Bank (KTDB) has been established to collect comprehensive information of trauma patients, including demographic characteristics, accident information, transfer information, severity, and treatment results. Data used in the study are based on the KTDB. The dataset includes a comprehensive database of over 400 distinct data elements for each patient, encompassing demographic and epidemiological characteristics, pre- and post-arrival clinical conditions, anatomic and physiologic parameters of injury severity, procedures, complications, and survival outcomes. Patients with traumatic injuries between January 1, 2019 and December 31, 2021 were selected. The following cases were excluded from the study among the selected patients. Based on the patient selection criteria according to the RTC evaluation guidelines presented by the Ministry of Health and Welfare: 1) Cases involving drowning, biting, poisoning, penetration, burn, or unknown causes; 2) Patients under the age of 15 years; 3) Patients referred from an outpatient department; 4) Patients transferred from another hospital to our RTC more than 48 hours after the injury occurred; 5) Cases where the patient returned home after receiving emergency room treatment or died in the emergency room; 6) Cases where the final Injury Severity Score (ISS) was missing; 7) Patients whose systolic blood pressure or respiratory rate was coded as ‘999’ upon visiting our RTC; 8) Patients whose systolic blood pressure was coded as ‘-1’ and whose response at the time of visit was coded as ‘alert,’ ‘verbal response,’ or ‘painful response.’; and 9) Patients whose respiratory rate was coded as ‘-1’ and whose response at the time of visit was coded as ‘alert,’ ‘verbal response,’ or ‘painful response.’

From 2019 to 2021, 5,953 patients were coded in the KTDB through our institution. Of these, 4,045 patients were finally enrolled after screening according to the exclusion criteria described above. To calculate the Ps using the TRISS method, Revised Trauma Score (RTS), ISS, and age are required.2 We calculated the RTS using the patient’s systolic blood pressure, respiratory rate, and Glasgow Coma Scale (GCS) at the time of initial visit.3 Table 1 describes the coded values used to calculate RTS. The ISS was calculated from the Abbreviated Injury Scale (AIS) based on the final diagnosis at the time of discharge. AIS scores are categorized into six body regions, and the highest score in each region is identified.4 The scores range from 1 for minor injuries to 6 for injuries that are typically fatal. The ISS was calculated by summing the squares of the three highest AIS scores for the most severely injured body regions.5 If a patient has an AIS 6 injury, the ISS is automatically assigned a value of 75. The formula for calculating Ps by combining RTS, ISS, and Age is as follows (coefficients revised in 2010 are described in Table 2)6:

P s = \frac{1}{1 + e^{- b}}

b = b0 + b1{SBP(c)} + b2{RR(c)} + b3{GCS(c)} + b4(ISS) + b5{Age(c)}

Table 1

Indexed values associated with GCS, SBP, and RR were used to determine the variable breakpoints for calculating the revised trauma score

Value	RR (breaths/minute)	SBP (mmHg)	GCS
0	0	0	3
1	1–5	1–49	4–5
2	6–9	50–75	6–8
3	> 29	76–89	9–12
4	10–29	> 89	13–15

GCS = Glasgow Coma Scale, SBP = systolic blood pressure, RR = respiratory rate.

Table 2

TRISS regression coefficient

Database	Blunt trauma
	b₀	b (RTS)			b₄ (ISS)	b₅ (Age^a)
	b₀	b₁ (SBP)	b₂ (RR)	b₃ (GCS)	b₄ (ISS)	b₅ (Age^a)
NTDB	1.6494	0.4260	0.0095	0.6307	−0.0795	−1.6216

TRISS = Trauma and Injury Severity Score, RTS = Revised Trauma Score, SBP = systolic blood pressure, RR = respiratory rate, GCS = Glasgow Coma Scale, ISS = Injury Severity Score, NTDB = National Trauma Data Bank.

^aAge = 1 if the patient is > 54 years, otherwise age = 0.

After calculating Ps, appropriate medical evaluation was conducted using W-score and mortality O:E ratio. The formula for W-score and mortality O:E ratio is as follows: A represents the actual number of surviving patients, and D represents the actual number of deceased patients.

W s c o r e = \frac{A - \sum P s (N)}{N} * 100

M o r t a l i t y O : E r a t i o = \frac{D}{\sum (1 - P s)}

The patients’ outcomes were classified as survival or death based on the medical records at discharge. The patients were divided into two groups: the neurotrauma patient group and the non-neurotrauma patient group, and RTS, ISS, Ps, W-score, and mortality O:E ratio were compared between these groups. The W-score indicates the difference between the actual number of surviving patients and the expected number of surviving patients, with the expected number of surviving patients being the sum of the Ps values for each individual patient. A higher W-score indicates that more patients were saved than expected while a lower score indicates fewer patients were saved than expected. The mortality O:E ratio is calculated by dividing the observed mortality by the expected mortality. The expected number of deaths is calculated by subtracting 1 from the Ps of each individual patient and adding the results together. Ratios greater than 1.0 suggest that there are more deaths occurring than would be expected, while ratios less than 1.0 suggest that there are fewer deaths occurring than expected.

Analysis for performance of TRISS method

Using the conventional TRISS method, patients’ expected survival or mortality was classified based on their Ps with a 25% cutoff value. Patients whose prognostic score exceeded 25% were predicted to have the potential for survival, whereas those with a score of 25% or lower were predicted to be fatal. Furthermore, a 2 × 2 table was constructed, including actual surviving and non-surviving patients in the analysis. The utility and limitations of the TRISS were assessed by calculating its sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and misclassification rate (MR) with respect to the Ps. The sensitivity of the TRISS was defined as the proportion of patients who were accurately predicted to expire among the total number of patients who actually died. The specificity was defined as the proportion of patients who were correctly predicted to survive among the total number of patients who actually survived. The PPV represented the proportion of patients who were accurately predicted to die among the total number of patients predicted to die, while NPV was defined as the proportion of patients who were accurately predicted to survive among the total number of patients predicted to survive. The MR was calculated as the sum of the number of patients who died but were predicted to survive and the number of patients who survived but were predicted to die, divided by the total number of patients. While both PPV and NPV are reliable indicators of the value of TRISS, the MR, which considers both PPV and NPV, is the most appropriate measure of the overall usefulness of the score.

Ethics statement

This study was exempted from review by the Institutional Review Board of Pusan National University Hospital (IRB No. 2303-003-124), and the requirement for informed consent was waived. This study is an analysis of data that do not contain personally identifiable information. Since the data analyzed pertained to patients who have already completed treatment, there is no direct risk or benefit to individuals. Therefore, we have submitted a consent waiver and received approval for this study. We used Microsoft Excel 2016 for Windows (Version 2301; Microsoft Corp., Redmond, WA, USA) for statistical analyses.

RESULTS

Based on the information from KTDB, this study analyzed trauma patients using the TRISS method to calculate the probability of survival. From 2019 to 2021, there were 5,953 trauma admissions that met the inclusion criteria for KTDB. However, 1,908 patients were excluded based on the exclusion criteria established by this study. Out of the remaining 4,045 patients, there were 358 deaths, resulting in a mortality rate of 8.85%. The predicted number of deaths based on the TRISS methodology was 415.7, suggesting that there were fewer deaths than expected (Table 3). Among the 4,045 patients included in the study, 1,639 (40.5%) suffered neurotrauma. Of the total 358 fatalities, 263 patients were admitted due to neurotrauma, indicating that neurotrauma was a significant contributor to mortality, accounting for 73.5% of all deaths. Statistical analysis showed that age did not have a significant effect on the difference between neurotrauma and non-neurotrauma patient groups. However, GCS, RTS, ISS, and Ps were all statistically significant in distinguishing between these two patient groups. Notably, patients with neurotrauma had significantly lower GCS, RTS, and Ps values, as well as higher ISS values (P < 0.001), suggesting that the severity is higher in patients with neurotrauma. The W-score for the entire study population was 1.426 and the mortality O:E ratio was 0.861, indicating good results. However, for the neurotrauma patients, the W-score was −0.680 and the mortality O:E ratio was 1.044, indicating poor outcomes. This means that patients with neurotrauma have a lower survival rate than predicted by the TRISS method.

Table 3

Summary

Variables	Neurotrauma patients	Non-neurotrauma patients	Total
Number of patients	1,639	2,406	4,045
Actual deaths	263	95	358
Actual survivals	1,376	2,311	3,687
Average of age, yr	56.7	57.5	57.2
Average of GCS	11.4	14.3	13.1
Average of RTS	6.7596	7.4323	7.1598
Average of ISS	21.0	14.9	17.4
Average of Ps	84.6%	93.2%	89.7%
Predicted deaths	251.9	163.8	415.7
Predicted survival	1,387.1	2,242.2	3,629.3
W-score	−0.680	2.861	1.426
Mortality O:E ratio	1.044	0.580	0.861

GCS = Glasgow Coma Scale, RTS = Revised Trauma Score, ISS = Injury Severity Score, Ps = probability of survival, O:E ratio = Observed-to-expected ratio.

The performance of the TRISS method in the two groups is presented in Table 4. Although the MR of the TRISS method was low in the non-neurotrauma group (3.6%), indicating excellent performance, the MR in the neurotrauma group was high (13.3%). Specifically, among the actual survivors of neurotrauma patients, 5 patients were predicted to die, while 203 patients who actually died were predicted to survival.

Table 4

Performance of TRISS method

Variables	Neurotrauma patients	Non-neurotrauma patients	Total
Sensitivity, %	19.0	23.2	20.1
Specificity, %	99.6	99.4	99.5
Positive predictive value, %	90.9	62.9	80.0
Negative predictive value, %	86.6	96.9	92.8
Misclassification rate, %	13.3	3.6	7.5

TRISS = Trauma and Injury Severity Score.

Table 5 demonstrated subgroups based on age, GCS, and ISS, which are used in the TRISS method. The performance of the TRISS method was evaluated by subgroup, specifically for those aged 55 or older, with a GCS score of 8 or lower, and with an ISS of 16 or higher. Levels of consciousness below GCS 8 and anatomical damage above ISS 16 are indicative of severe trauma. In the subgroup with GCS scores below 8, the MR was found to exceed 35%.

Table 5

Performance of the TRISS method according to subgroups

Variables	Age ≥ 55			GCS ≤ 8			ISS ≥ 16
Variables	NS	Non-NS	Total	NS	Non-NS	Total	NS	Non-NS	Total
Sensitivity	20.7	25.0	22.0	21.9	50.0	26.5	19.8	27.8	21.7
Specificity	99.5	99.3	99.4	98.8	83.5	95.2	99.5	98.6	99.0
PPV, %	90.7	65.5	80.6	94.2	62.9	81.6	90.9	62.9	80.0
NPV, %	84.6	96.1	91.5	58.7	75.0	61.5	82.1	94.0	87.5
MR, %	15.1	4.5	8.8	37.4	28.5	35.6	17.5	7.1	12.8

TRISS = Trauma and Injury Severity Score, GCS = Glasgow Coma Scale, ISS = Injury Severity Score, NS = neurotrauma patients, Non-NS = non-neurotrauma patients, PPV = positive predictive value, NPV = negative predictive value, MR = misclassification rate.

DISCUSSION

Despite the lack of a solid foundation for trauma and injury research in the country, a few researchers have introduced and analyzed assessment tools such as AIS, ISS, RTS, and TRISS.1 7 8 9 10 11 The establishment of the first RTC in Korea in 2004 marked a significant milestone, and the foundation for trauma research and care has gradually improved over the past decade. As the trauma center system reaches maturity, it is essential to analyze optimized datasets to improve the quality of care and assess the suitability of evaluation tools.

Trauma quality assessment involves evaluating the quality of care provided to trauma patients, including the accuracy and completeness of documentation, the appropriateness of diagnostic and treatment interventions, and the timeliness of care. Trauma quality improvement involves using the results of these assessments to identify areas for improvement and implementing changes to improve the quality of care. Trauma quality assessment and improvement can help ensure that trauma patients receive the highest quality of care possible, leading to better outcomes and decreased morbidity and mortality. It can also help trauma centers and systems identify and address areas of weakness, improve resource allocation, and enhance the overall efficiency of care delivery. As such, trauma quality assessment and improvement should be a priority for all trauma centers and systems, and should be conducted regularly to ensure that patients receive the best possible care. To achieve this objective, periodic evaluations of individual RTC are conducted under the supervision of the South Korean Ministry of Health and Welfare. Although expert panel discussions of deaths and complications can help identify and resolve issues within the RTCs, its utility in comparing outcomes between trauma centers is limited. To overcome this limitation, more objective and reproducible methods have been proposed. Introduced by Champion et al.3 12 in the 1980s, the TRISS methodology is the most commonly used formula for calculating the Ps for individual trauma patients. The Ps of TRISS for trauma patients is calculated using a formula that includes the mechanism of injury (blunt or penetrating), age, anatomical injury, and physiological status. The widespread adoption of the TRISS has led to the identification of several significant limitations of the method in many studies.7 8 13 14

This study found that although the overall MR of the TRISS method was 7.5%, which may suggest good performance, the MR for neurotrauma patients was significantly higher at 13.3%. The low overall MR can be attributed to the fact that the majority of trauma patients have minor or moderate severity trauma, which is associated with low mortality rates. In cases of minor or moderate-severity trauma, survival predicting formulas may not be necessary. The usefulness of a survival predicting formula is most critical in cases of severe trauma, and in this context, TRISS failed to provide any significant contributions. This study concluded that using TRISS without appropriate adjustments for severity can be risky and lead to errors when evaluating and comparing trauma centers. Demetriades et al.14 reported that when using the TRISS method for comparative analysis, large trauma centers that handle a significant number of severe trauma cases may be at a disadvantage and may not be appropriately compared to small trauma centers. The study by Margulies and colleagues,15 which utilized the TRISS method to evaluate trauma center outcomes, found that small trauma centers had better outcomes than larger centers. This finding is somewhat surprising, as larger trauma centers are often thought to have more resources and specialized staff to provide better care to severely injured patients. However, several reasons could explain for this unexpected result. For example, smaller trauma centers may have fewer patients, allowing for more individualized care and attention to each patient. Additionally, smaller centers may have more direct access to local resources, such as emergency medical services, which could facilitate faster response times and more efficient care. It is important to note that this finding may not be generalizable to all trauma centers and populations, and further research is needed to better understand the factors that contribute to better outcomes in different types of trauma centers.

The MR of the TRISS method was approximately 35% in both neurotrauma and non-neurotrauma patients with a GCS score of less than 9. The GCS is a measure of a patient's level of consciousness, with lower scores indicating more severe injury. The high MR indicates that the predicted survival rates based on TRISS did not match the actual survival rates in these subgroups of patients with a low GCS score. As a result, the TRISS system was considered “practically worthless” in predicting outcomes for severely injured patients with a low GCS score. This underscores the limitations of the TRISS method in accurately predicting survival for patients with severe neurotrauma injuries and highlights the need for alternative prediction models that can provide more accurate prognoses and guide clinical decision-making.

In this study, it was found that the W-score and mortality O:E ratio results of neurotrauma patients were inappropriately measured and showed poor outcomes. It is difficult to explain the inconsistency of the W-score and mortality O:E ratio in patients with neurotrauma as solely due to inappropriate treatment. Firstly, this is due to the suitability of the coefficients of the TRISS method. Lane et al.16 found that applying new coefficients obtained from patients at 12 trauma centers in Ontario resulted in better outcomes than when using coefficients from the Major Trauma Outcome Study (MTOS) database. The MTOS database has coefficients that are publicly recognized, but Lane et al. note that they may not be appropriate for hospitals in other regions with differences in treatment methods, treatment times, and other factors. Secondly, there is no consideration for infection, sepsis, or pre-existing conditions. Morris et al.17 pointed out that the TRISS method may not adequately consider the effects of pre-existing diseases or medical history. This could result in an increased risk of complications or death depending on pre-existing conditions or medical history, even if the ISS value is good. Thirdly, the RTS only adopts systolic blood pressure, the GCS, and respiratory rate at the time of admission, and does not reflect changes in physiological parameters. After a craniofacial injury, brain swelling tends to progress during hospitalization, and the GCS tends to decrease further. Fourthly, since the ISS only uses the three highest AIS, it does not reflect the relative importance of damage to different anatomical zones of the body, and even with the same ISS score, there is a disadvantage that the mortality rate can vary greatly depending on the location of the injury. It is difficult to accurately predict survival rates when multiple injuries occur in one area, such as the head and neck.

The W-score and mortality O:E ratio based on the TRISS methodology are used for evaluating RTC in Korea. However, during the mortality panel discussion, experts’ opinions on the preventability of death in neurotrauma patients and the Ps of TRISS showed significant differences. It is believed that this difference resulted from an overestimation of Ps in neurotrauma patients, which had an inappropriate impact on the results of the W-score and mortality O:E ratio. Therefore, these study results demonstrate that the W-score and mortality O:E ratio based on the TRISS method may be inappropriately predicted in the case of neurotrauma patients, and caution should be exercised when using the TRISS method. In addition, it emphasizes the need for the development of more accurate and reliable methodologies and indices for the evaluation of the preventability of death in neurotrauma patients.

In this study, patients were included or excluded based on the RTC evaluation guidelines established by the Ministry of Health and Welfare. The TRISS method predicted a high probability of death in patients with neurotrauma caused by penetrating injuries,6 and the actual fatality rate in this patient population was also high.18 Thus, if such patients were included in the study, the MR would likely be lower. Furthermore, patients transferred after 48 hours were likely to have lower severity and stable physiological conditions, resulting in a high Ps calculated by the TRISS method and a high likelihood of actual survival. It is also possible that the exclusion criteria may have increased the MR. The total number of patients with penetrating injuries was 396, which represented 6.65% of the total study population. Out of these patients, only one had neurotrauma caused by penetrating injury. Two hundred thirty-four patients (3.93%) were transferred from other hospitals after 48 hours, of whom 62 patients (1.16%) had neurotrauma. Given the low frequency of patients meeting the exclusion criteria, their effect on the MR was expected to be insignificant. Nevertheless, a follow-up study that includes all patients with available data among those meeting the exclusion criteria is necessary.

The present study provides evidence that underlines the potential limitations and drawbacks of using the TRISS method for predicting outcomes in patients with severe neurotrauma and for comparing survival outcomes between different trauma centers. Our findings suggest that the use of the TRISS method for these purposes should be approached with caution, as it may not always provide accurate predictions of patient outcomes and could potentially lead to misinterpretation of survival rates between trauma centers. Specifically, our study revealed that the TRISS methodology demonstrated a high MR of approximately 35% in subgroups of patients with GCS scores less than 9, both in neurotrauma and non-neurotrauma patients. This suggests that the TRISS method may be less reliable in predicting outcomes for severely injured patients with low GCS scores. Overall, our findings suggest that while the TRISS method remains a valuable tool for predicting outcomes in trauma patients, its limitations and potential pitfalls should be taken into account when interpreting results and making clinical decisions. Clinicians and researchers should be aware of these limitations and consider using alternative methods to evaluate patient outcomes and compare the quality of care provided by different trauma centers. Nonetheless, our study emphasizes the need for ongoing evaluation and optimization of trauma centers to ensure that all patients receive the best possible care.