PDF
Citation
Print
INTRODUCTION
Chronic kidney disease (CKD) poses a major global health and economic burden because of its association with comorbidities such as diabetes and cardiovascular disease. In 2024, CKD affected 8.4% of Korean adults, with patient numbers increasing by 8.2% annually. In Korea, the average annual medical cost per patient reached 8.5 million Korea Won (KRW) in 2021 [1, 2]. CKD is classified into five stages based on albuminuria and the estimated glomerular filtration rate (eGFR) [2]. eGFR accuracy is influenced by serum creatinine (Cr) variability, mainly due to measurement imprecision and calibration bias. Even a 0.12 mg/dL (10.61 μmol/L) Cr bias can cause a 12% error in GFR estimation, potentially altering CKD staging [3]. To ensure accuracy, the National Kidney Disease Education Program (NKDEP) recommends Cr measurement with <8% imprecision and <5% bias at 1.00 mg/dL (88.42 μmol/L) Cr concentration [4].
All in vitro diagnostics (IVDs) used in clinical laboratories must be registered or approved by regulatory authorities. However, the registration/approval process focuses on documenting correlations with existing IVDs rather than evaluating assay accuracy from a traceability standpoint [5–7]. The quality of IVD products must improve for better Cr results. To ensure accurate test results, additional evaluation and certification are necessary.
The International Federation of Clinical Chemistry (IFCC) and the Cholesterol Reference Method Laboratory Network (CRMLN) collaborate with assay manufacturers and clinical laboratories to standardize HbA1c and lipid assays [8, 9]. However, no internationally recognized reference laboratory network exists for Cr. In 2017, Korea established a national certification process for Cr measurements under the Academia-Government Collaboration for Laboratory Medicine Standardization in Korea (KR-STDZN) initiative inspired by IFCC, National Glycohemoglobin Standardization Protocol, and Centers for Disease Control programs [10]. The Korean program differs from other programs in terms of specimen type, certification process, criteria, methods, and costs. This initiative represents Korea’s first effort toward Cr assay standardization. The primary goal of the program is to ensure that analytical systems (instruments, reagents, and calibrators) meet quality standards, and the secondary goal is to improve measurement accuracy to support reliable clinical decision-making.
Cr results can vary significantly in function of the analytical system used, including the instruments, reagents, and calibrators, individually or in combination, developed by different manufacturers [11]. Cr assay accuracy is influenced more by the manufacturer than by the method used, even when the method is traceable to a reference method [12]. Biases in Cr results obtained using major instruments and reagents differ according to the manufacturer [13, 14]. This variability highlights the need for standardization efforts to ensure consistent and accurate measurements across different analytical systems, with a focus on Cr assay manufacturers. IVD products must meet rigorous analytical performance specifications to ensure that measurements are sufficiently accurate and precise for use in clinical laboratory tests.
We comprehensively analyzed IVD certification for Cr assays in Korea over 7 yrs (2017–2023) to gain insights into the effectiveness of the certification process. Our findings highlight the importance of standardized evaluation methods, particularly for assays that lack an international reference network, and offer a potential model for establishing similar certification processes in other assay categories.
METHODS
This study was exempt from institutional review board (IRB) approval (IRB No.: AMC 2024-0524). This study was conducted in accordance with the principles of the Helsinki Declaration and its amendments.
IVD certification for Cr standardization
The certification process is summarized in Fig. 1. The Korea Disease Control and Prevention Agency (KDCA) guides the annual schedule for the certification program. A certificate is issued for each analytical system that successfully meets the acceptable performance criteria based on the combination of instrument model, reagent lot, and calibrator lot. The entire process is usually completed within 8 weeks. The certification program is based on voluntary participation.
We used commutable frozen serum specimens in the clinically meaningful range of 0.3–5.0 mg/dL, including critical decision points at 0.6 mg/dL (53.05 μmol/L) and 1.0 mg/dL (88.42 μmol/L), as secondary reference materials. Two specimens were prepared at 12 concentrations each, making a total of 24 specimens. The Cr concentrations were classified into five or six range groups to cover the clinically relevant concentration range. Separate specimen sets were prepared for each analytical system, and manufacturers applying for multiple models or lots received additional sets to ensure thorough performance evaluation. To prevent cheating, the order of the 24 specimens was randomized. Supplemental Data Table S1 lists the concentration ranges and numbers of specimens corresponding to each range from 2017 to 2023. All specimens were tested for stability and homogeneity in accordance with the requirements stated in the ISO 35:2017 guidelines [15].
Acceptance criteria for IVD certification
We evaluated accuracy, imprecision, total error (TE), and linearity. The criteria for acceptable performance were the minimum analytical performance specifications of the NKDEP Laboratory Working Group for bias, CV, and TE [4]. The criteria for certification are bias≤5.1%, imprecision≤3.2%, and TE≤11.4%. Details of absolute percentage bias (absBias%) and TE calculation are provided in Supplemental Data Text S1. We estimated linearity or nonlinearity using the College of American Pathologists (CAP) linearity evaluation criteria [16]. The linearity limit was set to 3.75%, representing 25% of the TE target (15%).
Statistical analysis
We used SPSS for Windows (v.18.0; SPSS, Chicago, IL, USA), Microsoft Office Excel 2019 (Microsoft, Redmond, WA, USA), and MedCalc v.23.0.6 software for Windows (MedCalc Software, Ostend, Belgium) for statistical analyses. Normality was assessed using the Shapiro–Wilk test. Annual accuracy results for Cr concentrations were compared among groups using one-way ANOVA followed by Bonferroni post-hoc tests. Statistical significance was set to P<0.05.
RESULTS
Participation in the IVD certification program
The numbers of participating reagent manufacturers from 2017 to 2023 were 10, 14, 18, 19, 18, 18, and 16, respectively. The numbers of participating IVDs, depending on combinations of the reagent lot number, calibrator lot number, and instrument, were 14, 53, 166, 80, 95, 81, and 87, respectively, over these years. The distribution of participating IVDs based on the Cr assay type is summarized in Table 1.
Acceptable rates based on IVDs and assay types
Fig. 2 shows the percentage of IVDs that met the analytical performance specifications for bias, imprecision, TE, and linearity from 2017 to 2023. The acceptable rates for bias ranged from 54.7% to 85.3%, whereas imprecision rates were consistently high, ranging from 88.7% to 100.0%. Acceptable rates for TE and linearity ranged from 58.5% to 97.5% and from 76.8% to 96.6%, respectively. The percentage of IVDs meeting the performance specifications for each assay type is shown in Supplemental Data Fig. S1. Over the 7-yr period, the acceptable rates for bias were the highest for the rate-blanked compensated kinetic Jaffe method (KJ-RB) and enzymatic method (ENZ), whereas the kinetic Jaffe method without compensation (KJ-NC) showed lower acceptable rates.
Bias, imprecision, and TE
Over the 7 yrs, the average absBias%, CV%, and TE% were 4.6%, 1.37%, and 7.3%, respectively. The absBias%, CV%, and TE% for all participants according to the Cr assay type are summarized in Table 2. The mean bias% range over the 7 yrs was 2.3%–12.4%. The mean CV% range was 0.6%–3.1%, and the mean TE (%) range was 3.5%–17.1%. ENZ methods consistently showed the lowest average CV% and TE% among all assay types. The KJ-NC method showed the highest mean bias (12.4%) and TE (17.1%) in 2018 and the highest CV (3.1%) in 2017 because of its low acceptance rate. From 2019 to 2023, absBias% for the KJ-NC and KJ-C methods tended to decrease, with KJ-NC bias reduced to approximately 4% in the 1.0 (88.42)≤Cr<2.0 (176.84) mg/dL (μmol/L) range and to approximately 10% in the 0.0≤Cr<1.0 mg/dL range, although further improvement is needed.
For comparison with results from domestic manufacturers, we analyzed the results for four Cr reagents from foreign manufacturers that are frequently used in Korea, including Beckman Coulter (Brea, CA, USA), Roche Diagnostics (Penzberg, Germany), Sekisui Medical (Tokyo, Japan), and Siemens Healthcare Diagnostics (Tarrytown, NY, USA) (anonymized as A, B, C, and D). These manufacturers were selected based on their participation with ≥10 reagents and calibrators over the 7-yr period. These four reagents are extensively used in tertiary and general hospitals, and three of them are used in closed reagent systems. In contrast, reagents from domestic manufacturers are primarily used in hospitals and primary care facilities, mainly in open reagent systems. Relative biases for IVDs from the four reagent manufacturers according to the target value ranges from 2017 to 2023 are shown in Fig. 3.
We established five target value ranges: Cr<1.0 (88.42), 1.0 (88.42)≤Cr<2.0 (176.84), 2.0 (176.84)≤Cr<3.0 (265.26), 3.0 (265.26)≤Cr<4.0 (353.68), and Cr≥4.0 (353.68) mg/dL (μmol/L). From 2017 to 2019, the IVDs from the four manufacturers showed a large variation in bias% in the 0.0≤Cr<1.0 range, with CV values of 10.3%, 13.6%, and 7.1%, respectively. However, starting in 2020, the CV decreased to 4%–6%. In the 0.0≤Cr<1.0 range, the 7-yr average absBias% was 3.4% for company A, 5.0% for company B, 9.6% for company C, and 3.3% for company D. Company C improved IVD accuracy since 2019 by resetting the calibration value according to the instrument model and calibrator and reagent lot numbers. The bias results for the four companies mostly met the acceptance criteria for bias (≤ 5.1%) over the 7 yrs. The relative biases for the four manufacturers fluctuated over the 7 yrs, showing variability across years. The mean bias% was 1.0% in 2017 and peaked at 4.6% in 2018. From 2019 onwards, the bias generally stabilized closer to zero, with values ranging between –1.4% and –0.5%.
The absBias% values by target value ranges for the three manufacturers using closed reagent systems were below the average for all manufacturers, particularly in the range 0.0≤Cr<1.0 (4.1% vs 5.6%).
Comparisons of bias according to the year and target concentration
As for the methods, the 7-yr average absBias% was 3.3% for ENZ, 3.7% for KJ-RB, 3.8% for KJ-C, and 7.2% for KJ-NC. Bias in the function of the assay type for the five target ranges are shown in Fig. 4. The absBias%, CV%, and TE% for all participants in the function of the Cr assay type, and target value ranges for each year are summarized in Supplemental Data Table S2. For all assay types, the average bias% was the largest in the 0.0≤Cr<1.0 range, at 7.3%, followed by the 1.0≤Cr<2.0 range, with 4.0%, and the 2.0≤Cr<3.0 range, with 3.3%. In particular, within the 0.0≤Cr<1.0 range, the absBias% of the KJ-NC method was 14.2%, which differed significantly from the 5.0% for the other assay types. In the 1.0≤Cr<2.0 range, the absBias% was 5.7% for KJ-NC and 3.5% for the other assay types, which was less than that of the <1.0 mg/dL range but still significant. In the >2.0 mg/dL range, the absBias% of KJ-NC was <5.1%, which was slightly higher than the 3.0% bias observed for the other methods.
We used ANOVA to evaluate the effects of year and target value ranges according to the Cr assay type. The effect of the target concentration on outcomes varied depending on the year, with a significant interaction between these two factors. The annual comparative analysis results according to the target value ranges for each Cr assay type are presented in the Supplemental Data Table S3. We found significant differences among the four methods for the target value ranges, except for 3.0≤Cr<4.0 (F=2.246, P=0.038) and Cr≥4.0 (F=1.288, P=0.261) for KJ-NC.
DISCUSSION
The NKDEP Laboratory Working Group, in collaboration with international organizations, has initiated a global effort for Cr standardization by making calibration traceable to the isotope dilution mass spectrometry (IDMS) reference measurement procedure [4]. To date, unlike for HbA1c or lipids, no international laboratory network exists for Cr. Therefore, Korea has developed its own unique evaluation and certification system.
Our findings indicated that over the past 7 yrs, bias had the lowest acceptable rate among the four items evaluated. In 2018, the lowest acceptance rate for bias evaluation was 54.7%, primarily due to three main factors: specimen, assay principle, and open reagent systems. First, the issues in terms of specimen may include the concentrations of the specimens used, specimen stability, and commutability. The number of specimens with concentrations <1.0 mg/dL used for evaluation was higher in 2018 (33.3%) than in other years (16.7%). Low Cr values tend to cause large relative differences. Studies have reported unacceptable external quality assessment results due to poor accuracy for specimens with concentrations <1.0 mg/dL [13, 17, 18]. However, we did not evaluate specimen commutability and long-term stability and, therefore, could not assess the influence of bias due to the specimen itself. Second, 14 products from a single manufacturer using KJ-NC showed an average positive bias of 0.2 mg/dL. Therefore, the manufacturer was recommended to correct its calibrations by automatically subtracting a constant factor or modifying the assigned values. Of the 18 manufacturers using KJ-NC, only one uses open reagent systems. Open reagent systems or Cr assays based on KJ-NC showed a larger bias than closed systems or assays using KJ, as shown in previous studies [16, 19]. Open and closed systems use differently calibrated Cr measurement procedures, even when they use the same calibrator. In addition, a bias among calibrators with the same lot number may occur in clinical laboratories when reagents from different manufacturers have different metrological traceability and results. Therefore, errors in the manufacturer-defined working calibrator-assigned values may contribute to the low acceptance rates of KJ-NC.
The average bias in the results of the participating laboratories showed a decreasing trend over the 7-yr period; however, a small positive bias of 2%–3% remained. The fluctuations in bias by assay type were attributed to varying participating products and the non-compensated methods used by domestic manufacturers. To address these issues, increased manufacturer participation, additional training, and re-evaluation are being conducted to enhance product quality.
As the certification program is conducted for various analytical systems, the calibrator value assignments have the largest impact on the results. Additionally, the QC results of the laboratory, regardless of its specialization, may affect the outcomes. This aligns with the purpose of our certification program, which is to evaluate each reagent system to ensure that laboratories use high-quality products with high accuracy. Currently, legal regulations or regulatory requirements regarding using products that have not participated in the certification program are lacking. To increase the participation rate of manufacturers and promote the improvement of product quality, in 2019, we added the IVD certification program to the checklist of the Korean Laboratory Accreditation Program (KLAP).
A key strength of our program is its post-certification support, which provides expert consulting and correction services. When an analytical system produces unacceptable results in the first challenge, the manufacturer may apply for a second challenge with new specimens. In 2018, 14 products for an open reagent system using KJ-NC showed unacceptable bias, with an average of 0.17 mg/dL (14.9%). We provided calibration equations and recommended that the manufacturer to reassign calibration values to eliminate bias resulting from calibration drift. Consequently, the manufacturer adjusted the calibration values, applied a constant factor, and switched to KJ-C, which improved the accuracy to –0.04 mg/dL (–2.2%) in 2019. Another advantage is that, unlike other programs, our program does not charge manufacturer fees.
Currently, the minimum analytical performance specifications are applied because our program is in the early stages of implementation. As the accuracy of Cr analysis at the manufacturing level improves, more stringent acceptance criteria, such as desirable or optimal analytical performance specifications, will be applied to ensure high-quality products. In addition, we evaluated linearity, referring to the linearity evaluation in the CAP LN24 Survey [16]. The Clinical and Laboratory Standards Institute (CLSI) EP06-ED2 and CAP Survey linearity evaluation methods show some differences. CLSI EP06-ED2 uses weighted least-squares linear regression analysis [20]. We referred to the CAP survey at the start of our program, but in the future, we will consider applying the CLSI method to validate and verify linearity.
The present study had some limitations. A major limitation is the limited participation, with only a fraction of products available in the market being evaluated. Participating manufacturers are primarily those whose IVDs are used in general hospitals and tertiary hospitals, with laboratories that are accredited by the KLAP. In contrast, clinics and hospitals use reagents from various manufacturers that are difficult to identify, making them less likely to participate in our program. In the future, integrating IVD certification requirements into the quality assurance evaluation system for national health screening institutions, particularly for hospitals performing NHSP, will be essential to increase participation rates and, ultimately, improve testing accuracy. Second, the current program is conducted only twice a year, which limits timely responses to changes in Cr reagent lot number, which typically change every 3–5 months. We are considering expanding the program to a quarterly schedule or implementing an on-demand certification process to address this limitation. Third, although the program includes specimens stored for up to 2 yrs, we used specimens that had only been evaluated for long-term stability up to 1 yr, and we did not evaluate the commutability of processed specimens. Therefore, we could not unequivocally demonstrate that errors were attributable to the specimen source. However, based on the results of the multicenter evaluation, we believe that we can partially address the specimen issue using verified specimens. In the future, studies on long-term stability are required.
The bias evaluation results suggest a need for continued efforts to improve accuracy, while the evaluation results for precision and linearity were relatively good. As reported previously, the mean bias% of accuracy-based proficiency testing results has not changed significantly since 2014. From 2014 to 2019, the average bias ranged between 0.3% and 3.2% [13], and in 2023, the mean bias% ranged between –0.5% and 2.8% [21]. The CAP survey indicated a range of –5% to 10% [16]. Therefore, the bias cannot be reduced to zero and tends not to decrease further, which is an inherent limitation of the Cr method.
Our certification program has contributed to the performance improvement of KJ-NC manufacturers’ products. Although the other assay types have also improved since the implementation of the program, the differences are not significant. The program has achieved its goal of enhancing the accuracy of KJ-NC. Additionally, the performance of the other assays has been maintained, and the purpose of the program is to monitor compliance with minimum performance requirements. Improvements in Cr assay accuracy can no longer be achieved solely by improving the performance of clinical laboratories, making a manufacturer-level certification program necessary.
We reported the first manufacturer certification program for Cr assays using secondary reference materials for evaluating measurement accuracy and precision. While we do not aim to rank performance and, therefore, do not restrict the use of the lowest-performing products, we encourage and support manufacturers’ efforts to correct demonstrated bias through continued participation in the program. Current acceptance criteria and evaluation methods may become more stringent depending on international regulatory requirements, national regulatory agencies, end-user intended use, and IVD manufacturers, which will lead to opportunities for quality improvement. As the accuracy of manufacturers’ Cr assays improves, the performance of clinical laboratories will also improve. This, in turn, will lead to more accurate GFR estimations and a reduction in the misclassification of CKD stages.
XML Download