PDF
Citation
Print
INTRODUCTION
It is imperative to identify patient deterioration as early as possible to prevent significant delays in diagnosis and treatment that might result in life-threatening consequences such as cardiac arrest, unanticipated intensive care unit (ICU) admission, and even death [1-5]. Thus, an increasing number of hospitals have implemented a rapid response system (RRS) to detect early warning signs from patients and use preemptive measures to prevent major adverse events from occurring [6].
An RRS consists of two components: the afferent component, which is for early recognition of clinical deterioration in a patient, and the efferent component, which is the process of timely therapeutic intervention [7]. The afferent component of RRS can be largely divided into two different types based on its triggering method: a single-parameter-based activation and a scoring system-based activation [8,9]. A single-parameter system is activated when at least one parameter exceeds a predefined threshold. This system is generally easier to implement in hospitals because it only requires a certain number of simple criteria to be defined. However, alarm fatigue could be a potential problem when a single-parameter system is used because even a single value out of its normal range can trigger the RRS, potentially causing an unnecessarily large volume of false-positive activations [10-14]. In contrast, RRS activation through a scoring system is dependent upon an accumulation of scores that reflect the severity of the physiological abnormalities, and it is triggered only when it exceeds a certain score. This type of system avoids the high false-positive activation rate of the single-parameter system because it uses not only one parameter but a sum of scores from various parameters to be activated. However, scoring-based systems are more complicated than single-parameter systems; they are more difficult to implement and operate in most hospitals [15-19].
The efficacy of the RRS for the two different activation types has long been a topic of debate, but mostly in adult studies. Only a few pediatric studies on pediatric RRS can be found, and most of those focus only on the scoring-based RRS, commonly referred to as the pediatric early warning system (PEWS). Notably, a large-scale randomized clinical trial was conducted in Canada in 2015, and a comprehensive review of 66 studies with 27 unique identification tools was performed in the United Kingdom in 2019 to assess the effectiveness and validity of PEWS [5,8]. A few additional prospective studies estimated the sensitivity of RRS, but they were all based on different types of PEWS [16-18]. Reports on single-parameter RRS, especially for the pediatric population, are scarce. One Korean study used a single variable, which was heart rate (HR) [6], and another Korean study used changes in HR and respiratory rate (RR) as variables for their RRS [20]. They were both small-scale, single-center retrospective studies that are designed to evaluate the efficacy of the single-parameter RRS itself. Therefore, they were not representative of comprehensive evaluations of RRS activation based on a single-parameter system [6,20].
Therefore, as the institution that pioneered the Korean RRS both in adults and children, 11 years ago, we conducted a comprehensive evaluation including predictive performance for single-parameter-based RRS in children. Despite the retrospective nature of this study, our accumulated data and over ten years of experience will enable us to reach a meaningful conclusion.
MATERIALS AND METHODS
Ethics Statement
This study was reviewed and approved by the Institutional Review Board of Seoul National University Hospital (No. H-1904-161-1031). Informed consent was waived because of the retrospective nature of the study.
Study Setting and the RRS
This retrospective observational study was conducted at a single tertiary children’s hospital with 350 beds. Patients under the age of 18 years who were admitted to the general ward from January 2012 to August 2023 were included. In this children’s hospital, the single-parameter RRS was developed and implemented since 2010 under the name acute response system (ARS). Based on the system developed by Tibball et al. [2] in 2005 [6], thresholds by age group for each of the following seven parameters were set: systolic blood pressure (SBP), HR, RR, oxygen saturation measured by pulse oxygen saturation (SpO2), decreased perfusion, change in mentality, and urine output. The RRS was activated when at least one of the parameters exceeded the threshold. After ARS activation, a decision to either maintain or deactivate was made when a patient’s condition improved or when the patient was admitted to the pediatric ICU (PICU). The ARS thresholds by age group are shown in Table 1.
Table 1.
ARS activation criteria by parameter and age group
Data Collection and Preprocessing
The data set used in the analysis was provided by the data warehouse of the hospital information system. A patient’s age, sex, SBP, diastolic blood pressure, HR, RR, body temperature (BT), and SpO2 were measured during hospitalization, and the time of vital sign measurement along with the location of the patient were recorded. Vital signs measured outside of general wards, such as operating rooms or ICUs, were excluded from the analysis. The collected vital signs that fell into the category of non-physiologic range were excluded from the analysis because of the possibility of keystroke errors in the process of inputting the value of vital signs. Non-physiologic range was defined as follows: HR >300 beats/min or <30 beats/min, RR >120 breaths/min or <5 breaths/min, SBP >300 mm Hg or <30 mm Hg, and BT >42 ℃ or <30 ℃. Cases where vital sign values were recorded as strings or ranges rather than numbers were also excluded from the analysis.
Outcomes
The primary outcome of this study was to evaluate the performance of the ARS, which is defined as the predictive power of critical events. A critical event was defined as an unexpected PICU transfer or cardiopulmonary resuscitation (CPR) occurrence in a general ward. Unexpected PICU transfer was defined as PICU admission due to an acutely worsening clinical condition, excluding PICU admission for routinely planned postoperative care or planned procedures. A positive prediction of the ARS was defined as a case where a critical event occurred within 48 hours after the ARS was activated.
The secondary outcomes were compliance of the medical staff in activating the ARS, degree of contribution of each parameter to ARS activation, correlation between parameters and the occurrence of actual critical events, and the frequency of critical events by year. Compliance for ARS activation was evaluated by calculating the cases where ARS was not activated when it was required. In cases where compliance was low, we also analyzed the performance of predicting critical events when compliance was assumed to be 100%.
Statistical Analyses
To evaluate the critical event prediction performance of the ARS, all vital sign measurements were divided into an activation group and a non-activation group depending on the actual activation status of the ARS. The occurrence of critical events was compared for each group. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were analyzed for this evaluation.
Identifying the instances where ARS should have been activated—in other words, missed ARS activation—is necessary to calculate the degree of compliance with ARS. Among the seven parameters for ARS activation, SBP, HR, RR, and SpO2 are parameters that can be collected from past medical records. However, parameters such as perfusion, mental change, and urine output reduction are not easy to collect retrospectively. Thus, we evaluated missed ARS activation cases using only the four reliably available parameters (SBP, HR, RR, and SpO2). Possible underestimation due to the utilization of only four parameters in identifying missed ARS activation cases may be an important consideration during interpretation. Additionally, patients with underlying heart disease may have a baseline SpO2 at rest lower than that of normal children, opening the possibility that the recorded SpO2 value alone will not accurately reflect compliance of the ARS activation. Therefore, patients with underlying heart disease were excluded from the compliance analysis.
To analyze the relevance between critical events and triggered parameters, the following subgroups were created based on the activation criteria of each parameter: low SBP group, low HR group, high HR group, low RR group, high RR group, and low SpO2 group. Logistic regression analysis was used to analyze the association of critical events within each group. Results are expressed as odds ratios (ORs) with 95% confidence intervals (CIs). Categorical and continuous variables are presented as numbers (%) and medians (interquartile range), respectively. Data preprocessing and statistical analyses were performed using R software version 4.3.0 (R Foundation for Statistical Computing). P-values <0.05 were considered statistically significant.
RESULTS
Baseline Characteristics
During the study period, 94,219 patients were admitted 177,077 times, and a total of 20,493,885 vital signs were measured. After applying the exclusion criteria, 7,663,326 vital sign measurements were included in the analyses (Figure 1). Patient age was 5 years (interquartile range, 1–11 years), and 58.1% were boys. Other baseline characteristics of each group are summarized in Table 2.
Figure 1.
Flowchart of the study. The acute response system (ARS) activation group indicates that medical staff activated ARS during vital sign measurements, regardless of whether the criteria for ARS were met. The ARS non-activation group refers to cases where medical staff did not activate ARS during vital sign measurements, regardless of whether the criteria for ARS were met. ICU: intensive care unit; ED: emergency department.
Table 2.
Characteristics of each group according to the ARS activation
| Variable | Total (n=7,663,326) | ARS non-activation group (n=7,539,338) | ARS activation group (n=123,988) |
|---|---|---|---|
| Age (yr) | 5 (1–11) | 5 (1–11) | 6 (2–10) |
| Age group | |||
| 0–<3 mo | 474,630 (6.2) | 470,387 (6.2) | 4.243 (3.4) |
| 3–<24 mo | 1,669,518 (21.8) | 1,645,913 (21.8) | 23,274 (19.0) |
| 2–<5 yr | 1,325,109 (17.3) | 1,298,835 (17.2) | 26,274 (21.2) |
| 5–<10 yar | 1,754,663 (22.9) | 1,718,786 (22.8) | 35,877 (28.9) |
| ≥10 yr | 2,439,406 (31.8) | 2,405,417 (31.9) | 33,989 (27.4) |
| Male | 4,454,132 (58.1) | 4,381,075 (58.1) | 73,057 (58.9) |
| Vital sign | |||
| SBP (mm Hg) | 104 (94–114) | 104 (94–114) | 103 (90–115) |
| DBP (mm Hg) | 60 (51–70) | 60 (51–70) | 60 (50–72) |
| HR (beats/min) | 108 (90–129) | 108 (90–129) | 118 (78–143) |
| RR (breaths/min) | 24 (20–30) | 24 (20–30) | 32 (24–46) |
| SpO2 (%) | 100 (98–100) | 100 (98–100) | 98 (95–100) |
| BT (℃) | 36.9 (36.5–37.3) | 36.8 (36.5–37.3) | 37.1 (36.6–37.8) |
| Abnormal ARS criteria | |||
| Low HR | 350,863 (4.6) | 312,870 (4.1) | 37,993 (30.6) |
| High HR | 233,251 (3.0) | 202,456 (2.7) | 30,795 (24.8) |
| Low SBP | 91,800 (1.2) | 80,180 (1.1) | 11,620 (9.4) |
| Low RR | 10,451 (0.1) | 8,206 (0.1) | 2,245 (1.8) |
| High RR | 105,918 (1.4) | 64,727 (0.9) | 41,191 (33.2) |
| Low SpO2 | 161,371 (2.1) | 149,619 (2.0) | 11,752 (9.5) |
| Decreased perfusion | 26 (0.0) | - | 26 (0.0) |
| Decreased urination | 0 | - | 0 |
| Mental change | 0 | - | 0 |
| Warda) | |||
| Neurology | 1,208,113 (15.8) | 1,194,031 (15.8) | 14,082 (11.4) |
| Cardiology | 1,020,942 (13.3) | 1,007,496 (13.4) | 13,446 (10.8) |
| Gastroenterology | 1,000,084 (13.1) | 993,421 (13.2) | 6,663 (5.4) |
| Nephrology/rheumatology | 934,375 (12.2) | 921,627 (12.2) | 12,748 (10.3) |
| Hemato-oncology | 1,027,024 (13.4) | 980,969 (13.0) | 46,055 (37.1) |
| Other pediatric diseases | 965,845 (12.6) | 938,405 (12.4) | 27,440 (22.1) |
| Psychiatrics | 87,178 (1.1) | 87,171 (1.2) | 7 (0.0) |
| Minor surgery | 1,297,579 (16.9) | 1,296,187 (17.2) | 1,392 (1.1) |
| Other locations | 122,186 (1.6) | 120,031 (1.6) | 2,155 (1.7) |
Values are presented as median (interquartile range) or number (%).
ARS: acute response system; SBP: systolic blood pressure; DBP: diastolic blood pressure; HR: heart rate; RR: respiratory rate; SpO2: pulse oxygen saturation; BT: body temperature.
a) Each ward and its corresponding department are follows: Neurology - neurology department and neurosurgery; Cardiology - cardiology department and thoracic surgery; Gastroenterology - gastroenterology department and general surgery; Nephrology - nephrology department and urosurgery, Other pediatric diseases - endocrinology department, pulmonology department, and infection department; Minor surgery - orthopedics and ophthalmology; Other locations – radiology and imaging unit, hemodialysis unit, endoscopy unit and etc.
Main Outcomes
The positive predictive power of the ARS for predicting critical events was excellent, with a specificity of 98.5% and an NPV of 99.6%. However, the sensitivity and PPV of the ARS were low, at 24.0% and 8.1%, respectively (Figure 2A). The compliance rate of ARS activation was 15.6%, which was very poor (Figure 2B). When the positive predictions were calculated with the assumption of 100% compliance, the NPV came out to be 99.6%, which is the same as for low compliance. Sensitivity, on the other hand, increased significantly, from 26.8% to 88.5%, while the specificity decreased from 98.5% to 38.5%, and the PPV decreased from 8.1% to 1.8% (Figure 2A and C). A subgroup analysis was performed to evaluate the association between occurrence of critical events and characteristics of ARS parameters. Univariable and multivariable logistic regression analyses showed that the occurrence of critical events significantly increased in most groups (low SBP group, high HR group, low RR group, high RR group, and low SpO2 group). However, the low HR group showed a significant decrease in the occurrence of critical events (Table 3).
Figure 2.
The annual changes in the number of critical events. The annual trends based on whether the acute response system (ARS) was activated within 48 hours of a critical event. Cardiopulmonary resuscitation (CPR) event (A) and unexpected pediatric intensive care unit (PICU) admission (B). ARS was divided into an ARS activation group and a non-activation group according to whether ARS was activated within 48 hours prior to the occurrence of critical events.
Table 3.
Association between ARS activation criteria and actual critical event occurrence
The incidences of CPR events and unexpected PICU admissions per 1,000 patient-days were 1.84 and 0.13, respectively. The annual incidences of CPR events and unexpected PICU admissions per 1,000 patient-days in the activated ARS group were 0.069 and 0.871, respectively, while in the non-activated ARS group, these rates were 0.063 and 0.965, respectively. There were 58 CPR events in the non-activated ARS group, with the highest occurrence (29.3%) in the cardiology ward and the cardiothoracic surgery ward. The frequency of CPR events in locations other than the general ward was significantly different, with eight CPR events in the non-activated ARS group and only one CPR event in the activated ARS group (Supplementary Table 1). There was no difference in the incidence of unexpected PICU admissions and CPR events between the non-activated ARS group and the activated ARS group. The reduction in occurrence of CPR events was higher in the activated ARS group, but the reduction in unexpected PICU admissions was higher in the non-activated ARS group. Meanwhile, unexpected PICU admissions showed minimal variation within the activated ARS group (Figure 3).
Figure 3.
Effectiveness of the acute response system (ARS) and compliance with ARS activation. The ability to predict critical events by activating ARS in the real ward (A) and calculated compliance of the system (B). The ability of the ARS to predict critical events (C). Success in prediction is considered a critical event, defined as an unexpected pediatric intensive care unit admission or cardiopulmonary resuscitation event, occurring within 48 hours of ARS activation. NPV: negative predictive value; PPV: positive predictive value.
DISCUSSION
In this study, we analyzed the performance of a pediatric single-parameter RRS in predicting critical events and clinical characteristics experienced during its activation. Prediction of critical events showed good specificity and NPV, while sensitivity and PPV were low. In operating the ARS, a single-parameter RRS, the authors expected that, because the system was activated with only one anomaly, ARS activation could increase the alarm fatigue of the medical staff, especially when the patient’s condition was not serious. Even if the workload of medical staff is high, this system was maintained with the idea that patients who are deteriorating should not be ignored, i.e., even if the PPV was low, the sensitivity was expected to remain high. However, PPV and sensitivity were low at 21.4%. There may be two reasons for low sensitivity. The first is that the activation criteria are inadequate, and the other is that compliance for ARS activation is low. We think that the second reason is more important than the first because the NPV was high at 99.6%, and we believe this is supported by the fact that ARS activation compliance of the medical staff was very low at 15.6%.
The ARS is activated when an abnormal value for each parameter is measured: the nurse in charge enters the abnormal value into the medical record first, and the attending doctor confirms it. Therefore, low compliance is due to the structural fact that, even if there is an abnormal value that is outside the criteria, it cannot be activated unless the nurse or doctor in charge confirms it. Considering that the purpose of the ARS is to detect and intervene in a patient’s deterioration at an early stage, it may be somewhat contradictory to require the primary care provider to confirm the ARS activation process. This is supported by the fact that when the ARS is simplified, which assumes 100% compliance with medical staff, the sensitivity increases dramatically (from 24.0% to 88.5%), even though the PPV becomes lower. The NPV remained unchanged at 99.6%, which is very high, suggesting that the medical staff's compliance with the application of single-parameter RRS may cause serious problems.
It is also interesting that the occurrence of critical events was significantly higher for abnormal parameters, whereas the occurrence of critical events was lower for HR below the criteria than for normal HR (Table 3). These results are consistent with a retrospective observational study in children that reported that RRS activation for bradycardia alone did not lead to adverse outcomes [6]. On the other hand, one adult study reported lower HR in the older age group compared with patients under 65 years old before cardiac arrest, suggesting that HR may be affected by other factors such as age as well as patient deterioration [21]. Even considering these results from the literature, it is difficult to fully explain how low HR was associated with lower occurrence of critical events than normal HR. Therefore, we think that it can be explained by the phenomenon of lower HR at rest, i.e., if the patient was not in a stressful situation, low HR may explain low occurrence of critical events. However, it does not explain why low RR was associated with high occurrence of critical events; thus, further analysis is needed. On the other hand, higher than normal HR and RR were associated with higher occurrence of critical events, a result consistent with previous literature [22].
On the other hand, our study did not show a significant difference in occurrence of critical events in the ARS activation group and the non-activation group. This was contrary to the results of a systematic review of RRS in children, which found that RRS introduction lowered incidence by predicting critical events [23]. Furthermore, the results differed from the results of Tibballs et al.’s study [2], which is the basis of our ARS. We suspect that low compliance may have contributed to this conflicting result. For effective RRS, it is important that the patient is cared for in the PICU, but if the patient is in the early stage of deterioration, they will not be electively admitted to the PICU due to the lack of free beds. This may also be one of the explanations for the above result. Several previous literatures have suggested that the effectiveness of RRS may depend on the medical environment in which it is performed [8,24,25].
This study has several limitations. First it was conducted in a single center. Therefore, the medical specific environment of this hospital, such as the availability of beds in the PICU, likely affects the results and limits generalizability to other centers. However, this limitation is currently unavoidable because few hospitals have introduced pediatric RRS, and moreover, it is the only center in Korea that has operated with a single parameter system for over 11 years. Second, there were almost no mental changes, perfusions, or urine output data among the ARS activators. We think it is more likely that this is due to low compliance, rather than that there were no cases with these characteristics. Seven parameters were needed to detect patient deterioration, which may have affected the outcome because our analysis relied on a half-system with only partial information. Lastly, in the process of calculating compliance, we used a simplified version consisting of four subdivided groups: SBP, HR, RR, and SpO2. Therefore, there was room for underestimation in cases where ARS should have been activated. However, as mentioned earlier, there were few cases of actual activation due to mental changes, etc., thus the bias effect is not likely to be significant, which suggests the conclusion that compliance was bad.
In conclusion, this study showed that prediction of critical events by the ARS was excellent, but the sensitivity and PPV were poor. Additionally, the medical staff’s compliance during the ARS activation process was also poor. There is a need for further studies of RRS that are more independent of subjective physician decision-making and consider a comprehensive set of factors.
HIGHLIGHTS
▪ The single-parameter acute reactive system has good negative predictive ability for the occurrence of critical events but has low positive predictive ability.
▪ There was no significant difference in the frequency of critical events based on whether acute response system (ARS) was activated.
▪ Development of trigger tools for improving the predictive ability of ARS is necessary, and there is a need for sufficient infrastructure in the intensive care unit to care for clinical deterioration that is not detected in advance.
XML Download