PDF
Citation
Print
INTRODUCTION
There is a persistent need for early identification of patients at risk of cardiac surgery-associated acute kidney injury (AKI) because of the increased need for kidney replacement therapy (KRT) and increased risk of in-hospital mortality [1].
Despite advances in the conceptual understanding of the pathophysiology and epidemiology of cardiac surgery-associated AKI [2], consensus definitions and staging of AKI remain based on relative changes in serum creatinine (sCr) concentration, a decline in urine output (UO) [3], and the provision of KRT [4].
Several prospective observational studies have assessed whether the use of kidney tubular damage biomarkers such as urine neutrophil gelatinase-associated lipocalin (NGAL) improves early kidney risk assessment when measured in addition to sCr concentration [5-8]. The performance of such biomarkers as predictors of subsequent changes in sCr concentrations is poor, which may be influenced by various confounders such as the choice of the consensus definition of AKI, clinical setting, or type of sample used to measure NGAL (urine or plasma) [9].
Patients testing positive for kidney tubular damage biomarkers may not fulfill the traditional AKI criteria and may still have worse outcomes compared to those with negative biomarker findings. This recognition led to the proposal that new AKI diagnostic criteria should equally include both glomerular function markers (sCr) and kidney damage biomarkers (e.g., NGAL or other injury biomarkers) based on thresholds; however, there is still no consensus of the appropriate thresholds for such assessments [10]. Multiple statistical methodologies have been suggested for determining the “optimal” cutoff values for such tasks [11]. Therefore, using multiple methodologies, studies have determined and reported study-specific optimal NGAL cutoff concentrations to define NGAL positivity or negativity [5-7, 12-15].
We hypothesized that differences in the methodology of NGAL cutoff value determination and the system chosen for AKI classification influence patients’ clinical AKI-phenotype group allocation and the associated risk of adverse events according to the proposed AKI matrix system [10, 16]. Using data from two independent prospective cardiac surgery study cohorts [17, 18], we aimed to explore the extent to which such changes in methodologies and classification systems affect the calculated risk of adverse outcome [15, 16]. We hypothesized that patient outcomes and the magnitude of the attributed risk of adverse events between clinical AKI-phenotype groups vary according to the methodology used for NGAL cutoff selection and the choice of classification system for sCr/UO-based AKI.
MATERIALS AND METHODS
Study design
We used the existing data of 200 patients at an increased risk of AKI from the Berlin study cohort of a randomized, multicenter study on cardiac surgery (NCT00672334) performed at the German Heart Center Berlin between May 2008 and January 2012 [17]. Urine samples were obtained upon patients’ admission to the intensive care unit (ICU) and were immediately centrifuged at 5,000 rpm (~ 3,075 g) and stored at −80°C. Following completion of patient enrollment, NGAL concentrations in urine samples were measured using badge analysis as described below.
The second study cohort included 103 consecutively enrolled adult patients undergoing open-heart surgery at the University Clinic Magdeburg, Germany, which aimed to explore postoperative clinical kidney risk assessment using urine sampled at ICU admission for each patient and NGAL measured individually within 60 minutes according to routine laboratory diagnostics [18].
In both cohorts, NGAL measurements were performed in the central laboratory at the Institute of Clinical Chemistry and Pathobiochemistry of the University Clinic Magdeburg, using a standardized clinical platform assay for urinary NGAL (ng/mL; ARCHITECT, Abbott Diagnostics, Abbott Park, IL, USA), which is a Conformité Européenne (CE)-certified biomarker for the diagnosis of AKI in Germany. The CE certification mark indicates conformity with the health, safety, and environmental protection standards for products sold within the European Economic Area. The measurement interval of the ARCHITECT urine NGAL assay is 10.0–1,500.0 ng/mL, with an imprecision of ≤10% total CV. When using an automated dilution procedure, the assay can report values of up to 6,000.0 ng/mL. Further data on assay performance and handling requirements used in this study are described elsewhere [19]. The sCr concentrations were measured via routine laboratory diagnostics using an enzymatic method standardized by isotope dilution-mass spectroscopy on a Cobas 8000 modular analyzer (Roche Diagnostics, Mannheim, Germany). Complete sCr and urinary NGAL data were available for 199/200 and 100/103 patients of the Berlin and Magdeburg cohorts, respectively. Patients with missing data were excluded from analysis.
Laboratory investigators were blinded to the sample sources and clinical outcomes. In the Berlin study cohort, Medicine Ethics Committee, Charité University, Berlin, Germany approved the study (ZS EK 11 654/07), and written informed consent was obtained from each patient [17]. In the study performed at the University Clinic Magdeburg, Germany, the local institutional Review Board categorized the study as an audit of current clinical practice and approved prospective data collection and use for the purpose of this study. Patients received written study information, but the need for written consent was waived by the ethics committee (Ethics Committee, University of Magdeburg, Case 49/13) [18]. Full study details have been described previously [17, 18].
Definition of sCr/UO-based AKI
sCr/UO-based AKI was defined according to the Renal risk, Injury, Failure, Loss of kidney function, End-stage (RIFLE) [3] or Kidney Disease: Improving Global Outcomes (KDIGO) classification system [4], considering increases in postoperative sCr concentrations from baseline (preoperative stage) and the decline in UO criteria in both cohorts. Both classification systems define AKI as an increase in sCr concentration by a factor of 1.5-times from baseline or a decline in UO to less than 0.5 mL/kg/hr for up to 6–12 hours. In contrast to RIFLE, the KDIGO classification additionally considers an increase of ≥0.3 mg/dL (≥26.5 µmol/L) within 48 hours as AKI stage 1.
Methodology of NGAL cutoff selection
We used five methods to derive candidate NGAL cutoff values for clinical AKI phenotypes. In both datasets, the area under the ROC curve (AUC) value was calculated separately for the dichotomous outcome measure sCr/UO-based AKI according to RIFLE and KDIGO classification criteria fulfilling at least RIFLE-risk or KDIGO stage 1, respectively. We chose candidate cutoff values based on study-patient ROC curve analysis [11, 20], where each point on the ROC curve corresponds to a potential cutoff value and is associated with a sensitivity and specificity value pair. Threshold selection represents a compromise between sensitivity and specificity (Fig. 1) [21]. Previously, to derive the NGAL and sCr/UO-based AKI matrix, there has been no preference of favoring either high sensitivity or specificity over the compromise for an optimal balance of both indices [22]. Similarly, for the present derivation of cutoff values, no weighting of sensitivity and specificity was applied.
To determine an optimal cutoff value to differentiate between patients with the condition (sCr/UO-AKI) and those free of the condition (AKI-free), various methods have been proposed aiming to narrow the gap to the point in the ROC space that mathematically maximizes the AUC to “perfect test prediction”; that is, the point at coordinates (0, 1) in the upper-left corner of the ROC space.
The first method is based on previous studies [5, 13] that selected the maximum Youden index [23] as the cutoff value, which maximizes the sum of sensitivity and specificity by maximizing the distance from the diagonal chance line (y=x) connecting (0, 0) to (1, 1) in the ROC space. The chance line itself approximates an AUC value of 0.5. The Youden index for a given point on the ROC curve is calculated as follows:
The second method is one that balances sensitivity and specificity in assessing when the test sensitivity is equivalent to the test specificity. This point on the curve is located on the line connecting the upper-left corner (0, 1) to the lower-right corner (1, 0) of the ROC space. Hypothetically, at the point where sensitivity equals specificity, the product of these two indices (sensitivity×specificity) is at its maximum [11]. In this study, we chose the point closest to the equivalence of sensitivity and specificity, referred to as sensitivity≈specificity.
The third method minimizes the distance of a candidate cutoff value to the point on the curve closest to perfection (0, 1). This cutoff value should correspond to the optimal cutoff value chosen from all potential points available on the ROC curve [24].
Mathematically, searching for the shortest radius originating from (0, 1), the square of the distance from each data point on the ROC curve to (0, 1) is first calculated as:
where TPF is the true-positive fraction and FPF is the false-positive fraction [24]. The distance D to (0, 1) of any given point on the ROC curve is then calculated as follows:
Derivation of the AKI phenotype matrix
Patients were allocated to the clinical AKI phenotype matrix groups derived from each candidate cutoff value of 2×2 contingency table cells from the ROC curve with regard to meeting or not meeting the criteria for any stage of sCr/UO-based AKI, separately for RIFLE and KDIGO (functional impairment, dichotomization).
NGAL positivity (+), indicating the presence of structural or tubular damage, or NGAL negativity (–) was determined according to whether the urinary NGAL concentration measured at ICU admission after cardiac surgery was above or equal (≥) or lower (<) than each candidate cutoff value, respectively.
In summary, the following four different combinations defining the patient groups could be distinguished considering the RIFLE/KDIGO and NGAL statuses individually for each candidate cutoff value (Fig. 2): NGAL(–)/sCr/UO(–), NGAL(+)/sCr/UO(–), NGAL(–)/sCr/UO(+), and NGAL(+)/sCr/UO(+).
Outcome measures
The predefined outcome measures were acute KRT initiation, in-hospital mortality, and the combination of KRT or in-hospital mortality.
Statistical analysis
The odds ratio (OR) with 95% confidence interval (CI) was calculated to assess the associated difference in the risk of developing an outcome measure between the patient groups derived from the corresponding AKI phenotype matrix. When there were groups with zeros (N=0), we introduced a corrective factor of 0.5 for all cells. The two groups of zeros were not corrected or compared with respect to systematic differences. We then calculated the magnitude of risk disparity between the methodologies used for cutoff value selection with the following formula:
Since the OR is a relative measure of risk, a value of 1.0 would correspond to the assumption of no difference in the magnitude of risk of developing an outcome of interest between the cutoff selection methodologies “a” and “b.” SPSS version 28.0 (IBM Corp., Armonk, NY, USA), R Environment for Statistical Computing [26], and Microsoft Excel 365 version 16.43 (Redmond, WA, USA) were used for statistical analyses.
RESULTS
In the Magdeburg cohort, 14 (14.0%) patients had AKI according to the RIFLE classification and 23 (23.0%) had AKI according to the KDIGO classification, whereas in the Berlin cohort, 24 (12.1%) patients had AKI according to the RIFLE classification and 59 (29.6%) patients had AKI according to the KDIGO classification. The detailed perioperative patient characteristics of both cohorts are presented in Supplemental Data Table S1 and in Supplemental Data Fig. S1. The ROC curves and associated AUC values of NGAL in predicting RIFLE- and KDIGO-defined AKI in both cohorts are presented in Supplemental Data Fig. S2. All derived cutoff values and their associated statistical metrics are listed in Table 1.
The cutoff concentrations calculated from the corresponding ROC curves differed according to both the methodology used and the choice of RIFLE or KDIGO classification, ranging from 10.6 to 159.1 ng/mL in the Magdeburg cohort and from 16.85 to 149.25 ng/mL in the Berlin cohort. In both cohorts, the findings obtained when using the sensitivity≈specificity and lowest distance methods were similar. In the Berlin cohort, using the RIFLE classification, the lowest distance method yielded the same cutoff value as the maximum Youden index. Lower cutoff values subsided with distinctly higher sensitivity for sCr/UO-based AKI (RIFLE and KDIGO), as opposed to specificity improvements for higher cutoff values.
The fraction of potential subclinical AKI (AKI stage 1 S) shifted from 2% to 33% and from 10.1% to 26.1% for the KDIGO classification in the Magdeburg and Berlin cohorts, respectively (Table 2). With increasing cutoff values, more patients were eventually allocated to the NGAL-negative group.
When the highest cutoff values were chosen, more patients with adverse events (i.e., those who were more likely to have higher NGAL concentrations) were reassigned to the NGAL-negative group, which resulted in groups with NGAL-negative status having more events than those with NGAL-positive status (OR <1.0; Table 2 and Table 3, Fig. 1C).
The proportion of patients with kidney impairment additionally identified by urinary NGAL only in relation to the proportion of patients diagnosed as having AKI by conventional sCr/UO-based criteria differed by up to 8.33–16.49 times and 2.46–2.6 times between the lowest and highest candidate cutoffs for the RIFLE and KDIGO classifications in the Berlin and Magdeburg cohorts, respectively (Supplemental Data Fig. S3 and S4). The distribution of patients according to the derived cutoff values allocated to the NGAL/sCr/UO groups and their outcomes are shown in Table 2. The ORs illustrating the differences in the risk of developing adverse events between these groups are provided in Table 3.
Finally, the magnitude of the difference in the calculated risk (fraction of ORs) for adverse events between the group distributions varied considerably when the cutoff concentration was changed within the RIFLE or KDIGO classification (up to 18.33- and 16.11-times risk difference, respectively). This variation in the calculated risk was even more prominent when comparing the various cutoff methodologies between the RIFLE and KDIGO classification systems, reaching up to a 12.79-times difference in the Berlin cohort and up to a 25.67-times difference in the Magdeburg cohort for KRT initiation. As an example, such distinct changes in calculated risk resulted from an approximate doubling of the NGAL cutoff value from 79.3 to 149.3 ng/mL and the change of the NGAL cutoff value from approximately 10 to 160 ng/mL, respectively (Fig. 3, Supplemental Data Fig. S5A–D).
DISCUSSION
Using two independent cardiac surgery cohorts, we assessed the extent to which the methodology for NGAL cutoff value selection and AKI classification system affected patients’ clinical AKI-phenotype allocation and the associated risk of KRT initiation or in-hospital mortality.
We found an increased risk of KRT initiation and in-hospital mortality for patients testing NGAL-positive. This increased risk for adverse events found between clinical phenotype groups varied considerably in magnitude (up to a 25.7-times difference) according to the underlying sCr/UO classification system of AKI and the NGAL cutoff selection methodology applied.
At ICU admission, NGAL positivity carried more prognostic information over NGAL-negative findings, regardless of whether the kidney function acutely declined (positive RIFLE/KDIGO criteria), confirming the findings of a previous study [5]. However, multiple methods of determining attributed “optimal” cutoff values to define biomarker positivity have been suggested [11]. These thresholds were derived from ROC curves to predict sCr and UO-based AKI [5, 12, 13].
Previous studies on NGAL predicting adverse outcomes in various AKI settings selected certain cutoffs based on published values, manufacturer recommendations, or calculated cutoff values from the study population [5-7, 12-15]. Considering a variety of confounders such as the AKI classification system and the collection timing of samples for NGAL testing, a recent meta-analysis derived summary ROC curves and NGAL cutoff values for the prediction of sCr-based AKI [9], showing high variability in the data among the included studies, which may be closely related to the present findings on the risk disparity for adverse events.
Considering that higher NGAL concentrations likely reflect more severe kidney injury than lower concentrations, the associated risk of adverse events and specificity will steadily increase as the measured concentration increases [27]. Although cardiac surgery-associated AKI is common, severe adverse complications such as KRT initiation or mortality are less frequent [28, 29], which seems to be consistent with the lower frequency of high NGAL concentrations in the present cohorts.
Given the small samples investigated in these cohorts with relatively few adverse events, our data indicate that there may be study-specific “sweet-spot thresholds” where the risk prediction metrics (Table 1) will be the highest, resulting in favorable allocation of patients according to such specific NGAL cutoff values that more distinctively differentiate groups of relative risk of adverse events [30]. It is therefore reasonable that the applied cutoff value selection methods may identify the same cutoff values as “optimal” in a small cohort but identify different cutoff values as “optimal” in larger cohorts [24]. This may also explain the high level of heterogeneity of previously reported cutoff values [9]. Even a small shift in the cutoff concentration (ng/mL) may ultimately lead to a potentially meaningful clinical difference in outcome distribution and consecutive risk disparity, as the respective cutoff 2×2 table is subject to change (Figs. 1 and 2) [31].
Extending conventional AKI classification criteria to a clinically applicable AKI phenotype matrix-based system representing different pathophysiological phenotypes of AKI (Fig. 2) is desirable [32]. Characterization of clinical AKI phenotypes in such a matrix-based system is directly dependent on the cutoff value chosen for dichotomization [33]. In such a matrix-based framework, a biomarker may be of additional prognostic value only if it clearly differentiates the incremental risk of adverse events within the derived matrix-based groups rather than specifically showing good diagnostic performance for sCr/UO-based AKI according to a high AUC value [34].
We found that the statistical methodology for NGAL cutoff value selection may be a relevant factor potentially leading to inconsistent findings in patients’ phenotype allocation and in the calculation of the attributed risk of adverse events among different clinical phenotypes of AKI. We consider that the magnitude of such confounding factors will likely depend on the variability across patient cohorts and that calculations of risk differences may be affected by cohort sample size variations. Complementarily, in the aforementioned recent meta-analysis on NGAL cutoff values [9], we demonstrated that the underlying meta-analysis approach may also lead to considerable differences in findings for discriminatory estimators such as the AUC or derived cutoff values [35].
Many caveats remain to be resolved to more accurately define AKI phenotypes [32]. NGAL was previously associated with mortality in cardiogenic shock [36], but concentrations and discriminative ability may also be influenced by systemic conditions such as sepsis [37] or variations in urine flow [38], potentially confounding the findings. We did not correct the outcome data for the presence of septic conditions as influencing factors since supplemental data were not recorded [17, 18]; however, including C-reactive protein, a surrogate parameter for sepsis, did not improve the risk assessment of AKI phenotypes in a previous analysis of this cohort [5]. In the present study, the risk parameters and cutoff values were derived from and were dependent on the investigated cohorts; therefore, they may not be generalizable. Nevertheless, the internal validity of our findings is strengthened by the similarities in risk variability within two independent, well-defined cardiac surgery cohorts with a consequent timing of biomarker sampling. We do not preclude the conclusion that other biomarkers for AKI risk prediction may also be affected by changing biomarker threshold selections [39].
Changing cutoff concentrations or threshold methodology may also change the previous pathophysiological narrative and associated understanding of AKI phenotypes such as subclinical AKI [16, 33]. We suggest considering potential confounders [9] and emphasize that great care should be taken when establishing a cutoff concentration for clinical use. The variability in risk in our findings highlights the complementary prognostic relevance of kidney functional parameters and kidney injury biomarkers and the necessity to interpret both in conjunction with the clinical context. Further studies are needed to assess the potential clinical relevance of the cutoff value selection methodology [40] and to outline the associated risk disparity for the clinical phenotypes of AKI.
In summary, using two independent cardiac surgery cohorts, we confirmed our hypothesis that the magnitude of the attributed risk of adverse events varies according to the methodology used for NGAL cutoff selection and the classification system of sCr/UO-based AKI. Further studies are needed to address alternative methods and evaluate the advantages and limitations of different approaches, in addition to those based on the ROC curve and dichotomization.
XML Download