PDF
Citation
Print
INTRODUCTION
Noninvasive ventilation (NIV) is one of the respiratory therapies used for treating patients with acute respiratory failure (ARF) [1]. The goal of NIV is to improve pulmonary ventilation and oxygenation [2,3] while preventing the need for invasive mechanical ventilation (MV) and associated increases in complication rates, hospitalization time, and healthcare costs [4,5].
Despite the known benefits of NIV, inappropriate use or prolonged application can contribute to treatment failure, increasing the risk of injuries associated with heightened diaphragmatic effort and the need for MV, which in turn leads to increased intra-hospital morbidity and mortality [5,6]. The failure rate of NIV therapy is approximately 30%, with 47% of these patients’ experiencing mortality. In contrast, the mortality rate for patients in whom ventilatory therapy is successful is 10% [7].
The failure rate of NIV therapy is related to delays in clinical decision-making regarding treatment failure and the need for MV [8]. Consequently, methods to predict NIV failure and facilitate early facilitate early switching to MV when indicated. Initially, these methods were poorly standardized and depended on the healthcare team's experience or were related to parameters such as vital signs, level of consciousness, arterial blood gas, and severity scales of disease [9]. However, in 2017, a scale that aggregated parameters including Heart rate, Acidosis, Consciousness, Oxygenation, Respiratory rate (HACOR) was developed with the aim of better predicting NIV failure [9].
The HACOR scale aims to predict the risk of NIV failure in a standardized, objective manner with good accuracy. The scale score ranges from 0 to 25 points. Scores of 5 or higher indicate a risk of failure greater than 80% (with 72.6% of sensitivity and 90.2% of specificity), regardless of the patient's age, disease severity, or diagnosis [9]. However, in 2022, an updated version of this scale was introduced because the authors identified that respiratory failure was not solely due to respiratory causes but also involved broader, extrinsic factors beyond pulmonary issues. In the updated HACOR version, the patient's clinical diagnosis and the severity of the disease using the Sequential Organ Failure Assessment (SOFA) scale are also scored [10]. In this new proposal, the score is categorized into four cut-off points, with patients scoring ≤7 considered at low risk of NIV failure and those scoring >14 considered at very high risk of failure [10]. With the addition of these variables, the updated model demonstrated greater predictive capability compared to the previous version [10].
It is important to determine whether this updated version of HACOR has similar or superior clinimetric properties to the original version, since it requires more information from the patient and, therefore, is more laborious. Thus, the objective of this study was to compare the reliability, measurement error, interpretability, convergent validity, and predictive validity of the original and updated HACOR scales in patients with respiratory failure admitted to the emergency room. This is because the most common cause of admissions to the emergency department is respiratory in origin and the most commonly used treatment is NIV [11].
MATERIALS AND METHODS
This prospective observational study with repeated measures was approved by the Ethics Committee of Research Involving Human Beings at Hospital Santa Marcelina de Itaquera (CAAE 48137421.0.0000.0066), and the patients or their guardians signed the informed consent form.
Participants
Hospitalized patients in the emergency department of the hospital aged over 18 years who had hypoxemic or mixed respiratory failure with at least one of the following signs were included: decreased inspired oxygen concentration with the need for supplemental oxygen therapy, alveolar hypoventilation with decreased peripheral oxygen saturation (SpO2) or arterial partial pressure of oxygen (PaO2) ≤60 mm Hg, increased respiratory effort, use of accessory muscles, and paradoxical breathing at rest [12,13].
Participants were classified based on the type of ARF and its corresponding disorder following a review of electronic medical records, clinical investigation, and thorough medical assessment of the possible origin of the ARF [14,15]. Patients with contraindications for NIV were excluded [16], including: cardiac or respiratory arrest, inability to cooperate (decreased level of consciousness, drowsiness, agitation, and patient refusal), hemodynamic shock (systolic blood pressure<90 mm Hg), complex cardiac arrhythmias, encephalopathy, severe gastrointestinal bleeding, upper airway obstruction, inability to protect the airway (ineffective cough and swallowing difficulties), abundant airway secretions, abdominal distension, nausea or vomiting, acute myocardial infarction, facial trauma and deformity, recent postoperative status of facial surgery, upper airway or esophagus, claustrophobia, or immediate need for MV.
Procedure
Patient selection for this study occurred immediately upon the patient’s arrival at the emergency department. After the patient or their guardian signed the Informed Consent Form, the following data were collected: age, sex, weight, Simplified Acute Physiology Score (SAPS) 3, SpO2, and history of previous pulmonary or cardiovascular disease. Additionally, the cause of hypoxemic or mixed ARF was classified as either pulmonary or extrapulmonary, related to either mixed disturbances or hypoventilation [14,15].
One hour after the initiation of NIV, the HACOR scale was administered by the first evaluator (test), using the Brazilian-Portuguese version translated simply. At this time, SpO2 was also measured (comparison measure for convergent validity). After 10 minutes, the scale was re-administered by a second evaluator (inter-rater reliability test). After another 10 minutes, the HACOR scale was applied again (retest). The clinimetric properties evaluated included reliability, measurement error, ceiling and floor effects, convergent validity, and predictive validity. Analyses followed the COSMIN (Consensus-based Standards for the selection of health Measurement Instruments) criteria [17].
Instruments
The HACOR scale ranges from 0 to 25 points and reflects the sum of each score of the scale variables: heart rate 0 (if ≤120 bpm) to 1 point (if ≥121 bpm); pH 0 (if ≥7.35) to 4 points (if <7.25); Glasgow Coma Scale 0 (if =15) to 10 points (if ≤10); arterial oxygen partial pressure to fraction of inspired oxygen ratio (PaO₂/FiO₂) 0 (if >201) to 6 points (if ≤100); and respiratory rate 0 (if ≤30 bpm) to 4 points (if ≥46 bpm). Higher scores indicate higher risk of NIV failure. A score of 5 or more indicates the need for orotracheal intubation [9].
The updated HACOR scale incorporates both the original HACOR scale and the SOFA for disease severity. Additionally, it uses the patient’s diagnosis to calculate the score as follows: Original HACOR score+0.5×SOFA+2.5 if pneumonia is diagnosed, –4 if cardiogenic pulmonary edema is present, +3 if acute respiratory distress syndrome (ARDS) is present, +1.5 if immunosuppression is present, +2.5 if septic shock is present. The result is categorized into four cutoff points: patients scoring ≤7 are considered at low risk of failure, 7.5–10.5 moderate risk, 11–14 high risk, and >14 very high risk for NIV failure [10].
SpO2 was collected through bedside monitoring and portable peripheral oximetry, as well as heart rate. Respiratory rate was also collected via bedside monitoring; however, when visualization was not available on the monitor, respiratory rate was counted manually. The Glasgow Coma Scale was also assessed at bedside [18]. pH and PaO2 were obtained through arterial blood gas analysis conducted by the physician and sent to the hospital laboratory for analysis; evaluators then consulted the results directly from the unit’s exam system. The FiO2 was recorded from the mechanical ventilator monitor.
Intubation Decision and Follow-up
During the patient’s clinical course, the attending physician made care decisions regarding orotracheal intubation for MV without knowledge of the HACOR score, thereby maintaining independence from the study. Patient follow-up after HACOR retest was conducted via electronic medical records, collecting the following data: need for MV, discharge date, intubation date, and date of death.
Statistical Analysis and Description of Clinimetric Properties
Patient data were subjected to descriptive analysis. The properties were analyzed as described below. Data analysis was performed using IBM SPSS statistical software version 22 (IBM Corp.), with the confidence level set at 5% (P<0.05).
Reliability refers to the degree to which the assessed instrument is free from measurement error [17]. Reliability was tested both intra-rater and inter-rater using the intraclass correlation coefficient (ICC) subtype of absolute agreement for single measures, where the variance of measurements in each participant, rather than the mean, was considered (ICC(2,1)), along with its corresponding 95% CI. The classification adopted was: less than 0.40=poor reliability; between 0.40 and 0.75=moderate reliability; between 0.75 and 0.90=substantial reliability; and greater than 0.90=excellent reliability [19].
Measurement error relates to the absolute error of measurement, which refers to how close two or more repeated measures are to each other [20]. Measurement error was tested using the standard error of measurement (SEM=test score minus retest score, divided by SD) and the minimum detectable difference with 90% confidence (minimal detectable difference 90= test score minus retest score, divided by √2×SEM) [18,21]. SEM was considered very good if <5% of the total score, good if ≥5% and <10%, questionable if ≥10% and <20%, and poor if >20% [17].
Ceiling and floor effects show the scale's ability to discriminate across different conditions (severities) of patients through scoring. They were tested by calculating the percentage of patients who achieved the maximum (ceiling) and minimum (floor) scores, and were considered present if 15% or more of individuals reached the maximum or minimum score in the assessment [21]. This indicates whether the scale has limited validity.
Convergent validity was assessed by Pearson’s correlation between the HACOR scale score and SpO2. The correlation was classified as follows: r<0.30 indicates a weak correlation; r≥0.30 and<0.60 indicates a moderate correlation; and r≥0.60 indicates a strong correlation [21]. The a priori hypothesis considered that the correlation between the HACOR scale and SpO2 would be negative (in direction) and moderate (in magnitude) (r≥0.60), as the direction of the scores has an inverse relationship with the patients' clinical condition, and oxygenation is one of the components of HACOR. This measure is the score of the evaluated instrument compared with other similar measures, along with hypotheses of the concepts being measured, to reduce the chance of bias [22].
Predictive validity relates to a test's ability to discriminate a future outcome [19]. The receiver operating characteristic (ROC) curve was plotted, and the area under the curve was calculated to determine the HACOR scale’s discriminative ability of the need for MV. The cut-off value for the HACOR scale was determined through ROC curve analysis and defined as the point with the maximum Youden index (i.e., sensitivity + specificity – 1) [23].
RESULTS
Ninety patients were considered eligible during the study period, and 60 completed the protocol (Figure 1). The main reasons for exclusion of patients were refusal to participate in the study (n=11), intolerance to NIV (n=7), immediate need for MV (n=5), and claustrophobia (n=3), among others. The demographic, anthropometric, and clinical characteristics of the patients are summarized in Table 1.
For scale applications, the median (interquartile range [IQR]) score was similar both in the test and retest phases (3 [1.0–5.0] vs. 3 [0.25–6.0], P=0.56)of the HACOR scale, as well as for the updated HACOR (5 [3.0–8.87] vs. 5.25 [3.0–8.87], P=0.38).
The average duration of NIV use among patients was 1.08±0.27 hours. Eighteen of the 60 patients (30%) required MV. Twenty-five patients (41.7%) included in the study had died by the end of the follow-up period.
The clinimetric properties of the original and updated versions of the HACOR scale are compared in Table 2. Intra- and inter-rater reliability were excellent for both versions of the scale, with an ICC ≥0.95 according to COSMIN [17,21,22]. Measurement error was considered very good in both versions (original HACOR version: SEM=0.25 and DMD=0.085; updated HACOR version: SEM=0 and DMD=0). Convergent validity with SpO2 was moderate for both HACOR scales (original: r=–0.39, P=0.002; updated r=–0.42, P=0.001). Predictive validity was adequate for both versions (original: ROC=0.73; updated: ROC=0.78), being 5% upper in the updated HACOR version, and both versions had the same high level of sensitivity (0.92). The original version of HACOR showed a ceiling effect that was not observed in the updated version.
DISCUSSION
This is the first study to compare the clinimetric properties of the HACOR scale created in 2017 [9] with the updated HACOR scale from 2022 [10]. Both versions of the scale demonstrated adequate intra-rater and inter-rater reliability, measurement error, convergent validity, and predictive validity for detecting NIV failure after one hour of treatment in patients with ARF admitted to an emergency department. However, the updated version showed better performance across all properties, including the ability to predict the need for MV. The original version exhibited a ceiling effect that was not detected in the updated HACOR.
The results indicated that the HACOR scale can be applied either by the same professional or different professionals without compromising accuracy, as reliability was excellent in all scenarios and the instrument error was very low, demonstrating high applicability in clinical practice. Furthermore, comparing HACOR scale performance with single-variable assessment methods that comprise the scale, both HACOR versions were superior for detecting the need for invasive MV (92% sensitivity), whereas respiratory rate had 71% sensitivity, and the APACHE II scale achieved 86% sensitivity for predicting the same outcome [24-26]. This demonstrates that individual variables provide specific information that, when combined, becomes superior to their value alone for detecting NIV failure. However, most studies agree that diagnosis can influence the clinical decision for orotracheal intubation [24-32].
Despite both scales having adequate clinimetric properties and the slight superiority of the updated HACOR version in terms of predictive validity, the application of this version in an emergency department setting may face some barriers, as reported by Duan et al. [10]. For example, the need for multiple laboratory tests and almost immediate processing to calculate the SOFA scale, as well as to complete the patient's diagnosis [33], may compromise the usability of the updated HACOR scale. However, it improves predictive capacity since ARF is not limited to ventilatory issues alone [27-32], which may explain why this version was superior to the previous one. Conversely, the original HACOR version is practical and quick to apply [9], as it does not require a wide range of laboratory tests for risk stratification of NIV failure, making clinical decision-making more expedient. Therefore, choosing the best version of the scale to be used may depend on the routine and structure of each emergency or intensive care unit.
This study had some limitations, with the first being that the number of patients was lower than the number determined by COSMIN [17,22]. Nevertheless, the methodological rigor was high and meticulous, allowing for the acquisition of realistic and reproducible data. Additionally, many patients scored low on the scales, possibly because NIV may have achieved a high success rate in this group of patients despite the observed initial severity and high mortality rate due to extrapulmonary complications.
In conclusion, both versions of the HACOR scale have adequate clinimetric properties for detecting NIV failure, including reliability, measurement error, and convergent validity. However, the updated version demonstrates greater predictive validity and the absence of a ceiling effect compared to the original version.
KEY MESSAGES
▪ Objective assessments of the success or failure of noninvasive ventilation (NIV) are necessary to avoid delay the indication for mechanical ventilation and increasing the risk of complications.
▪ The two versions of the Heart rate, Acidosis, Consciousness, Oxygenation, Respiratory rate (HACOR) scale have adequate clinimetric properties to detect failure of NIV.
▪ The updated version of HACOR is more complex than the original version; however, it shows superior predictive validity and no ceiling effect.
XML Download