DISCUSSION
SAPS II is one of the most commonly used prognostic scoring systems in critically ill patients, but a revised scoring system, SAPS 3, was devised for several reasons. First, the SAPS II score was developed from a database collected in the early 1990s, and there have been significant changes in the prevalence of major diseases, diagnostic approaches, and therapeutic modalities since that time.
1-
3 Second, previous prognostic models did not take into account the clinical milieus of different regions of the world, being developed mostly from clinical data of European and North American origin.
1,
2 Third, many reports suggested that SAPS II has poor predictive power in different populations, limiting its usefulness.
7
The SAPS 3 scoring system was developed to enhance prediction power by overcoming these deficiencies. The SAPS 3 has the following unique characteristics. First, SAPS 3 is composed of 20 variables gathered within one hour of ICU admission.
2 So SAPS 3 is not affected by the Boyd and Grounds effect,
2 thus in theoy it should have reduced risk of overestimated prediction about the mortality rates. Second, SAPS 3 reflects the chronic health status and the conditions before admission to ICU which can influence long-term prognosis of these patients.
2 Finally, SAPS 3 features customized equations that were developed to consider regional differences in disease distributions, genetic factors, and therapeutic behaviors.
To our knowledge, this is the first study exploring the validation of general SAPS 3 or its customized equation for Australasia in patients of Korea. External validation is essential before routine application of any model in a group of patients different from the one originally used for model development. So far there have been only a handful of studies of the external validation of SAPS 3 and its customized scores, with mixed results. In an external validation study of a general intensive care population in Europe, SAPS 3 and its customized model for Central and Western Europe were more discriminative and had better calibration compared to Acute Physiology and Chronic Health Evaluation II (APACHE II), but were not significantly better than SAPS II.
9 An Austrian validation study found that the SAPS 3 admission score overestimated hospital mortality but that the customized equation showed excellent calibration and discrimination.
1 Validation of the SAPS 3 score in Brazil, and, in particular, its customized equation for Central and South American countries, was successful in critically ill patients with cancer.
8
In our cohort, the general SAPS 3 admission prognostic model and the Australasia SAPS 3 had good calibration. However, the SAPS II model exhibited poor calibration. Performance of SAPS II in our cohort was similar to other reports:
10,
11 acceptable discrimination but lack of calibration. In most cases, the lack of calibration was often accompanied by an underestimation of mortality in low risk patients and an overestimation in high risk patients.
7 The discriminative power of general SAPS 3 and Australasia SAPS 3 were better than that of SAPS II, but no single model had an aROC value exceeding 0.8, a threshold generally considered to indicate good discrimination. In our cohort, both general SAPS 3 and Australasia SAPS 3 revealed a lower discriminative power rather than that in the original SAPS 3 study (aROC 0.848, 95% CI 0.98, 1.02; aROC 0.839, 95% CI 0.85, 0.99 in original SAPS 3 study).
7 A pattern of good calibration with poor discrimination is one frequently found when existing severity scoring systems are evaluated on populations different from the ones for whom these models were originally developed.
12,
13
One of the most important findings of our study was that the customized equation for Australasia (Australasia SAPS 3) did not result in better predictive power than the general SAPS 3 equation. The regional equation was developed for more precise estimation from a more homogeneous case mix. However, in our cohort, Australasia SAPS 3 as a local or regional equation did not improve the predictive power of the original prediction model. Several factors may have potentially contributed to that fact. First, although the customized equation for Australasia was derived from patients in Australia, India, and Hong Kong, these patients may differ from our cohort in terms of genetics, disease distribution, or other factors. Australia, which contributed more than one-third of the patients in the original cohort, is a multi-ethnic country with a large population of European descent. Other than the fact that they are geographically close to each other, there is no intuitive reason to combine data from Australia, India, and Hong Kong to formulate a customized equation for patients of Asian descent due to differences in genetics and medical and social systems. Australia is more like European countries than other countries in the region. It would be interesting to see how the equation might have differed if only patients of East Asian descent were included.
Another explanation might be that our cohort was sicker and only involved medical patients, while the original cohort included less sick patients and more diverse patient samples. In the original cohort of SAPS 3, the median SAPS II score was 28, and ICU mortality was 12.7%, which is significantly different from our cohort (SAPS II mean of 44). In addition, 25% of our cohort was made up of cancer patients, while in the original SAPS 3 cohort this fraction was only 10%. Finally, different patterns and quality of ICU care might have resulted in these discrepancies. However, our observed mortality was similar with the predicted mortality from all three scores, which suggests that the results of this study did not result from poor intensive care quality.
Our study has several limitations. It is a retrospective analysis of prospectively collected data, but we made every effort to validate the data as thoroughly as possible. Second, the data are from a single center with a relatively limited number of patients, which limits the generalization of our findings.
In conclusion, the SAPS 3 admission prognostic model had good calibration and modest discriminative power when applied to ICU patients in Korea. However, its customized equation for Australasia did not improve predictive power. Therefore, a new prognostic model customized for Korean patients is deemed necessary.