The severity of illness in critically ill patients is measured using a variety of scoring systems, including acute physiologic and chronic health evaluation (APACHE) Score, mortality prediction model (MPM) and simplified acute physiology score (SAPS) to predict clinical outcome. In addition to mortality prediction power, these severity scoring systems are also used to evaluate the quality of critical care based on the comparison of predicted mortality to actual mortality. Also, the effectiveness of care bundle practiced in the intensive care unit (ICU) setting is also assessed by comparing mortality predicted after application of bundles against actual mortality.[1,2] Among severity scoring systems, simplified acute physiology score 3 (SAPS3) was developed in 2005 to upgrade the then existing scoring systems. SAPS3 is used to determine disease severity on ICU admission using 20 clinical variables.[3,4] Although severity scoring systems have been improved, no standards have been established for assessment methods and severity ratings. More importantly, the available scoring systems tend to be too complicated and time consuming. In fact, even in US, scoring systems were used only 10-15% of ICUs.[1]

One of the reasons that severity scoring systems are not commonly used in the ICU can be the need for human resources and time. Some researchers describe SAPS3 scoring is less complicated and time consuming than other scoring systems because it is based on smaller number of variables. [5] However, SAPS3 still takes significant time for inexperienced users to complete. To improve the validity of the predicted mortality, variables should be measured within one hour after ICU admission. Any delay in data collection can overestimate the mortality.[4] Given the importance of timing, ICU nurses can assess severity of illness if ICU physicians are not in a position to perform it in the timely manner. Recently, there are many studies led by nurses, and in some studies, SAPS3 were used as a severity score. We therefore explored in this study if there were any difference in the inter-rater reliability (IRR) of SAPS3 scores assessed by ICU nurses.

Sum score and predicted mortality of one patient can vary depending on raters. Strand K[6] investigated IRR of simplified acute physiology score 2 (SAPS2) and SAPS3 performed by doctors. We evaluated the IRR of SAPS3 performed by ICU nurses using intraclass correlation coefficient (ICC) and compared the results with the previously reported IRR of doctor-assessed SAPS3 score. As a secondary aim of this study, we evaluated the ICC of SAPS3 subscores or boxes to find out what variables cause raters to agree and disagree.

This study was exempted the review of the Medical Research Ethics Committee of the Inje University Ilsan Paik Hospital (IRB No. IB-2-1310-042).

When five nurses independently assessed SAPS3 scores in 26 patients, ICC of SAPS3 score was 0.89 (95% confidence interval (CI) was 0.82 to 0.95) and ICC for the mean predicted mortality was 0.91(0.85 to 0.96). Among SAPS3 subscores analyzed in this study, ICC of SAPS3 box I, II and III was 0.90 (0.82 to 0.95), 0.54 (0.35 to 0.73) and 0.95 (0.91 to 0.97), respectively, showing the highest IRR in box III (Table 2). Thus, the IRRs of SAPS3 sum score and predicted in-hospital death were high whereas the IRR of SAPS3 box II was quite low. The ICC value of SAPS3 box II and corresponding variables are presented in Table 3.

In this study, we found trustworthy ICC of nurse-assessed SAPS3 scores and predicted mortality in ICU patients. Strand K et al[6] established the IRR of doctor-assessed SAPS2 and SAPS3 scores. Their study design was compared with that of this study in Table 4.

As far as the results of both studies are concerned, the ICC value of SAPS3 box II was equally low in this study as that calculated based on doctor-assessed scores in the previous study.[6] However, nurse-assessed SAPS3 score in this study exhibited a high IRR of 0.89, while doctor-assessed SAPS2 and SAPS3 scores exhibited 0.84 and 0.80. respectively.[ 6]

According to our analysis of SAPS3 subscores, the ICC value of SAPS3 box II was 0.54, showing the lowest IRR for variables dependent on diagnostic information. Coincidently, the ICC of SAPS3 box II was also 0.54 in Strand K et al’s findings.[6] The source of disagreement among raters appears to include that variables are open to different interpretations and that some complex of variable definitions are not easy to remember. Training can focus on such sources of disagreement to improve the reliability among raters. The ICC of SAPS3 box I was slightly lower at 0.90 than 0.94 reported by Strand K et al.[6] However, the reliability of SAPS3 box III was 0.95, showing a significant difference from 0.73 in their study. Overall, this study demonstrated higher ICC of SAPS3 sum score than the study of Strand K. et al[6], which conducted with doctors (0.89 vs 0.80).

As the latest upgraded version of SAPS2, SAPS3 offers a high explanatory power of 50% in SAPS3 box I, which is followed by box II (22.5%) and box III (27.5%).[4] Given smaller explanatory power of Box II to the total sum score, the wide disagreement among nurse raters on box II (where clinical judgment is required) did not affect the overall IRR of SAPS3 score significantly. In our study, IRR of box III was higher (0.95) than previously reported level among doctor raters (0.73), contributing to high ICC of SAPS3 sum score.

Currently, to determine appropriate nurse staffing in the ICU, surgical intensive care unit optimal mobility score (SOMS) or workload management system for critical care nurses (WMSCN) scoring systems are commonly used for disease severity assessment.[9] However, these tools are less useful to quantify severity of illness and mortality prediction for research purposes. Therefore, SAPS3 can be used for disease severity scoring, by nurses, in studies led by nurses. When SAPS3 scores are provided by multiple raters, IRR across the raters can pose an issue that needs to be resolved. This study demonstrated consistency among nurse raters.

The nurses studied had been working in the ICU for mean length of 7.8 years (range 3-11 years). Participant’s relatively long working years as critical care nurses pose a limitation for this study. However, over 5-year work experience may be common among ICU nurses in Korea. Park and Gang[10] stated that 37.9% of ICU nurses in their study had working years of 5-10 years, making up the largest group. It can be said that the participants of this study represents the population to be studied. However, further study is needed to investigate the IRR of new nurses in the ICU. This study ascertains the consistency across nurse raters but does not establish the IRR between nurses and doctors. If both nurses and doctors assess SAPS3, the ICC should be measured again to determine consistency between them.

Despite benefits of severity scoring systems, scoring is not widely performed in the ICU largely due to constraints related to human resources, time and finance: human resources and time for manual data collection and financial resources for a new system that will combine electronic medical records and scoring tool for automatic process.[1] This study aimed to identify the reliability of ICU nurse-assessed severity score and demonstrated high ICC of nurse-assessed SAPS3, which is in agreement with the ICC value calculated from doctor-assessed SAPS3 scores. This study therefore suggests that ICU nurses are able to assess severity of illness in ICU patients and that the resulting scores can be reliable.[6]

In conclusion, this study established the reliability of nurse-assessed SAPS3 scores. Although the established IRR of SAPS3 sum score is satisfactory, compared with the IRR of doctor-assessed SAPS3 score, the overall IRR among can be further improved through better understanding of diagnosis-related variables evaluated for box II.