PDF
Citation
Print
INTRODUCTION
Standardization is the process of obtaining equivalent clinical results from different measurement procedures using a calibration method that is traceable to a reference measurement procedure (RMP) or certified reference materials (CRMs) [1]. Owing to issues concerning cost, availability, and commutability of RMPs and CRMs, standardization cannot always be achieved in real-world practice. As an alternative to standardization, harmonization encompasses the concept of achieving equivalent results without a CRM and/or RMP, which implies that results are comparable irrespective of the time and place [2]. Harmonization is used for numerous clinical laboratory measurement procedures that do not have a technically reasonable or commutable reference material and can be a practical solution for calibration traceability for various measurands that would otherwise be non-harmonized [3]. Standardization and/or harmonization ensures the comparability and interchangeability of laboratory results and prevents physicians’ confusion and misinterpretation of laboratory results [1, 3, 4].
In large healthcare centers with multiple instruments, patient samples may be exposed to various laboratory test settings. If the instruments for a particular test are not harmonized, different measurement systems may yield biased results for the same sample, resulting in an increased risk of misinterpretation or malpractice. Therefore, periodic comparability verification is necessary to provide clear and harmonized results from different instruments, regardless of the approaches used [5]. However, global standardization or harmonization of clinical laboratory results is currently limited by cost and labor-effectiveness, as well as various technical and environmental challenges, and Korea is no exception. Step-wise protocols for standardization have been developed till date for only a few clinical tests, including cholesterol, creatinine, hormones, and glycohemoglobin [6–10]. More economic and simpler within-laboratory harmonization or comparability verification methods are required to provide convincing and reliable outputs from large laboratories using multiple instruments.
Comparability of results assures that the examination of a measurand is consistent within a laboratory system, even if different methods and instruments are used; however, there is currently no consensus on how to demonstrate such comparability. Laboratories should establish their own procedures to ensure results comparability within their individual systems. The clinical and laboratory standard institute (CLSI) provides guidelines (EP31-A-IR) for comparability verification within a single healthcare system; however, it does not suggest universally accepted criteria or specific methodology for each laboratory [5]. The CLSI guidelines present only limited examples of various situations and methodologies, for which the solutions involve infrequent and complicated processes that are unsuitable for application in specific routine measurements [5, 11]. We propose an easy and frequent periodic comparability verification method that is applicable to various clinical chemistry instruments for the harmonization of diverse tests performed at healthcare centers.
MATERIALS AND METHODS
Instruments and test items
Weekly comparability verification of various clinical chemistry tests using multiple instruments was prospectively performed at Seoul National University Bundang Hospital from January 2015 to June 2019. Four different types of instruments were used: Beckman Coulter AU5800 Chemistry Analyzer (Beckman Coulter, Inc., Brea, CA, USA), Abbott Architect Ci16000 Integrated System (Abbott, Abbott Park, IL, USA), two Siemens Dimension Vista 1500 Intelligent Lab systems (Vista1 and Vista2, Siemens Healthineers, Erlangen, Germany), and Ortho Vitros 5600 Integrated System (Vitros, Ortho Clinical Diagnostics, Raritan, NJ, USA). AU5800 was assigned as the standard comparative instrument as this is one of the main instruments of the laboratory used for largest number of clinical chemistry tests and is also a closed system that uses exclusive calibrators and reagents for clinical tests. Reference range setting and proficiency testing (PT), including accuracy-based PT performed by Korean External Quality Assessment Service (KEQAS) [12] and College of American Pathologists (CAP) [13], were also carried out on AU5800.
Three categories of clinical chemistry tests were evaluated for harmonization: (1) electrolytes, including sodium (Na), chloride (Cl), potassium (K), phosphate (P), and calcium (Ca); (2) liver panel (LP), including aspartate transaminase (AST), alanine aminotransferase (ALT), alkaline phosphatase (ALP), albumin, and total protein (TP); and (3) selected standardized test items, including total cholesterol and creatinine. Detailed information regarding the calibrators, reagents, average number of tests per year, imprecision during the study period, and comparative systems for each test item and instrument are shown in Supplemental Data Table S1. For the standardized test items, including total cholesterol and creatinine, commutable reference materials were produced and stored for a nationwide accuracy-based PT.
Sample preparation for comparability assessment
Residual serum samples were used to prepare sample pools from two to five patients for each biological material considered for comparability assessment. Residual samples were stored for seven days, which were used only during the storage period. Samples were pooled as larger volumes to enable adequate comparability verification between the five instruments. The pooled serum samples originated from residual samples of individuals who visited the hospital or outpatient clinic for healthcare check-ups with venous blood collected for various clinical tests in seven days. More than 40 samples for the initial comparison and 10–20 samples for the simplified comparison were prepared at different analyte concentrations spanning the lower and upper limits of the measurement range in accordance with the CLSI EP09-A3 guideline [12].
Comparability assessment protocol
A schematic diagram of the process used for verification of comparability, including initial comparison, weekly comparability verification, and simplified comparison, across the five different instruments is shown in Fig. 1.
Initial comparison and conversion action
The initial comparison was performed according to the CLSI guidelines using more than 40 residual samples [5]. The percent bias (PBIAS) between the comparative AU5800 system and each instrument was calculated. For electrolytes and LP, if the PBIAS for any instrument exceeded the comparability testing acceptance criteria, the results were converted using a normal linear regression equation (intercept, a; slope, b) to make them comparable to the AU5800 results. The relationship between the measured concentration (Cmeasured) and converted concentration (Cconverted) was as follows:
To standardize the results of all instruments to the international reference system, an additional comparison was performed for cholesterol and creatinine using CRMs with target values measured using RMPs. When non-comparability existed, the results were converted to be comparable to the reference targets.
Weekly comparability verification
Comparability verification of the five instruments was performed each week by measuring LP, electrolytes, cholesterol, and creatinine levels in two pooled serum samples, and comparing the results. To evaluate the harmonized outcome of periodic comparability verification, we calculated the PBIAS of the weekly results between AU5800 and the other instruments and examined the trendline. If the PBIAS exceeded the acceptance criteria of agreement or comparability for two to four weeks, the following simplified comparison was performed.
Simplified comparison
If non-comparable results were observed for a specific test or instrument for a few weeks, a simplified comparison was performed using 10–20 of such non-comparable samples for electrolytes and LP or using commutable reference materials for cholesterol and creatinine to determine an appropriate conversion factor according to the linear regression equation. The converted and harmonized results from all instruments following periodic verification were reported to the clinicians rather than the original values. If subsequent results again failed to meet the acceptance criteria, a decision was made to either eliminate the conversion factor or change it to another value based on a new comparison evaluation.
Categorization of the results and statistical analysis
All results were categorized into three groups according to each analyte and instrument: (1) measured data without a conversion action for comparable results (non-converted results), (2) measured data before a conversion action for non-comparable results (measured data before the conversion action), and (3) converted results after the conversion action for non-comparable results (converted results after the conversion action). The number, average absolute PBIAS relative to the reference system, and percentage of non-comparable results (PNR) were analyzed for each category. Inter-instrument variation, expressed as the CV, was also calculated for each analyte according to the three groups before and after the conversion action. Significant differences between groups was evaluated according to the P-value based on the paired Student t test. Microsoft Excel 16.0 (Microsoft Corporation, Redmond, WA, USA) was used for data recording, comparability checking, and statistical analysis. MedCalc 19.1.7 (MedCalc Software Ltd, Ostend, Belgium) was used for all other analyses.
Weekly comparability verification and simplified comparison
Over the five years of the study period, approximately 432 weekly inter-instrument comparability verification results were collected for the three categories (electrolytes, LP, and cholesterol/creatinine) in pooled residual serum samples. We applied the Royal College of Pathologists of Australasia (RCPA) total allowable limit of performance goals to our database and then compared the weekly results from each instrument to determine whether they were within the allowable range compared with the AU5800 results (Table 1) [13]. Simplified comparison was performed for data that exceeded the allowable range of comparability (non-comparability data). For the within-laboratory harmonization of the clinical chemistry tests, converted results were reported to clinicians after the measured results were multiplied by the appropriate conversion factor obtained via linear regression analysis of the simplified comparison results (conversion action). The history of the conversion action is outlined in Supplemental Data Table S2. This research was granted review exemption by the Seoul National University Bundang Hospital Institutional Review Board (IRB No. X-2108-705-903).
RESULTS
Initial comparison
Following analysis of the initial comparison of more than 40 samples, initial conversion factors were applied to the instruments for specific items when the PBIAS between the comparative system and individual instruments was greater than the agreement acceptance criteria. Initial conversion factors were applied to 62% (31/50) of the test results from the various instruments, as shown in column “2015-02-03” of Supplemental Data Table S2.
Trendlines of weekly comparability verification
Fig. 2 shows examples of PBIAS trendlines for AST. Among the three test categories, verification of LP showed the most obvious changes from unconverted results, which may lead to non-comparability due to significant differences in calibrators or reagents. For example, converted AST results from multiple instruments showed good within-laboratory comparability until the summer of 2017 (in allowable limits, ±12%), whereas the original data before conversion action showed unacceptable PBIAS for several instruments, especially for the Vitros instrument in 2017 (−28.5%) and for the Vista instrument in 2017 (−19.2%) and 2018 (−22.7%). When the reagent was switched from one not containing pyridoxal-5-phosphate (P5P) to one containing P5P in March 2018 in all instruments, the original PBIAS of the Vista instrument was reduced to −20%. According to this PBIAS drop for AST tests, we calculated new conversion factors for each instrument to adjust the within-laboratory harmonization of AST results, and comparability was successfully achieved (allowable PBIAS±10%). As the original results for the Vitros instrument had become non-comparable with the reference instrument due to the P5P-containing reagent, the conversion factor was again modified to harmonize the AU5800 and Vitros results and maintain comparability.
Although the acceptable PBIAS ranges were narrower in the electrolyte category relative to those in the other clinical chemistry test categories, the converted results from all instruments reported to clinicians were in accordance with the acceptance criteria and comparable to the reference instrument; however, the original results for several tests from the Vista and Vitros instruments were outside the acceptable range and non-comparable to the AU5800 results. Serum calcium measurement from the Vitros instrument originally showed high PBIAS relative to the reference instrument (2%–10%), and the converted PBIAS oscillated within the acceptable range (−2% to+4%).
Unlike most tests that maintained within-laboratory harmonization and comparability with the AU5800 reference system, tests with an available nationwide accuracy-based PT program, including serum creatinine and cholesterol, showed slightly different patterns of comparability verification compared with other categories. Therefore, creatinine and cholesterol results were adjusted to the international reference values by recalculating conversion factors if non-comparable results were obtained.
Distribution of absolute PBIAS and PNR
The average absolute PBIAS (12.8%) and PNR (23.8% of measured data with conversion action, 17.5% of total results) from data before the conversion action were significantly higher than those from data after conversion action (4.3%, P=0.0005 and 4.6%, 2.5% of total results, P<0.0001, respectively) (Table 2). This tendency was observed for all instruments and test items, with only a few exceptions when group numbers were small.
Among the 432 comparability data points for each test item and instrument, approximately 58% were converted due to non-comparable results. The rates of conversion action differed according to the test item and instrument. In general, the rates of conversion of LP and electrolytes were lower for Ci16000 (27% and 11%, respectively) and higher for Vista (86% and 66%, respectively) and Vitros (79% and 81%, respectively). The rates of conversion action for cholesterol and creatinine were higher for Ci16000 (100% each) and lower for Vista (13% and 31%, respectively) and Vitros (13% and 0%, respectively).
Inter-instrument CV before and after conversion action
For all analytes, the CVs for both measured data without conversion action and converted results after conversion action were much lower than those for the measured data before conversion action (Table 3). The inter-instrument CVs of the converted results after conversion action were the lowest for electrolytes and higher for LP. We compared the ratio of inter-instrument CV to the allowable bias by European recommendations for each category [16]. Most ratios from the measured data without conversion action and converted results after the conversion action were <1. In contrast, ratios from original data before conversion action were >1, suggesting that the inter-instrument CV from the original data before the conversion action was not within the allowable bias.
DISCUSSION
The International Consortium for Harmonization of Clinical Laboratory Results (ICHCLR) was established to fulfill the recommendations of an international conference in 2011 convened to review the available infrastructure and challenges in achieving harmonization of results among different measurement procedures [3]. To achieve simple within-laboratory harmonization of clinical tests in a healthcare center, the CLSI EP31-A-IR guidelines suggest periodic comparability verification that can be applied to various circumstances, even in the absence of universally accepted criteria or specific methodology [5]. These guidelines suggest practical and general considerations, including which samples can be used for verification, frequency of assessment protocols, and examples of various acceptance criteria for comparability testing. However, the example regarding comparability verification of clinical chemistry tests in the appendix of the guidelines would require the preparation of specific replicate samples for multiple runs of each test on each device and specific acceptance criteria for each category. This methodology would likely be labor-intensive and expensive, making it difficult to follow frequently and only suitable for laboratories with few instruments from the same manufacturer [5, 11]. We developed a simple and frequent comparability verification methodology applicable to large healthcare centers with multiple clinical chemistry measurement instruments from different manufacturers.
In our laboratory, clinical chemistry tests were performed using five instruments from four manufacturers. To perform both cost-effective and labor-effective periodic comparability verification, we prepared pooled human sera from residual samples. Each comparability verification was performed weekly using the pooled sera and then by determining whether the results compared with the reference instrument (AU5800) were within the total allowable error (TAE) limits based on the RCPAQAP [15]. Although the comparability of our five instruments was monitored at relatively short intervals, this approach did not incur any additional costs other than limited labor and time for sample collection and pooling. The verified and corrected results from multiple instruments showed good comparability over the five years except for a few minor incidents that were promptly adjusted once non-comparability was detected. Therefore, we recommend this simple comparability verification methodology for laboratories in similar environments.
The average absolute PBIAS and PNR were much lower in the converted results than in the original data before conversion action. This enabled reporting more accurate and harmonized results to clinicians, even though we could not report all results within the acceptance criteria of TAEs. If weekly comparability verification had not been performed, our laboratory would have reported approximately 17.5% erroneous and non-harmonized results to clinicians; with its implementation, these errors were reduced to approximately 2.5% depending on the instrument. We found lower inter-instrument CVs in the converted results than in the original data before conversion action. Most CV ratios of the original data to allowable bias were >1.0, except for TP, potassium, chloride, and cholesterol, whereas the CV ratios of converted results to allowable bias were <1.0, except for phosphorus and creatinine. These results suggest that our weekly verification of comparability and appropriate conversion action resulted in successful intra-laboratory harmonization of the five instruments.
Serum electrolytes are important factors in many clinical decision-making situations, and their analysis in a healthcare center may be performed serially or periodically using different measuring instruments. Serum electrolyte tests should have a narrow range of acceptance criteria for laboratory comparability. According to the ICHCLR report of measurands, electrolytes such as sodium, potassium, and chloride should be adequately harmonized and maintained given their high medical impact [17]. In our weekly comparability verification, these three electrolytes showed relatively low PBIAS and PNR along with low inter-instrument variation, confirming their excellent harmonization status.
For LP items, which have wider allowable limits for comparability than electrolytes, comparability verification was successfully performed during the five years, except for intermittent events, including changes in reagents containing P5P. As P5P is a coenzyme of the AST and ALT reaction in the human body, it is included in the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) recommendations for measuring aminotransferase levels [18, 19]. However, several laboratories use a modified IFCC method assuming that P5P concentrations in patient serum are sufficient and therefore reagents without P5P can be used for AST and ALT tests [20, 21]. We discourage the modified IFCC method on the basis of its variable total error, which cannot be corrected through calibration [20]. As serum P5P levels in patients are often insufficient, serum AST and ALT levels tend to be lower when using the modified IFCC method, particularly in patients with poor health status, such as those with acute myocardial infarction [22]. After we changed reagents in February 2018 to meet IFCC recommendations, PBIAS trendlines of AST comparability verification changed remarkably. However, we immediately detected the event and successfully adjusted the comparability of the reported LP via our periodic comparability verification and conversion factor adjustments.
A high level of testing accuracy is clinically important for analytes such as creatinine and hemoglobin A1c (HbA1c). Accuracy-based PT is mandatory for a laboratory to determine the accuracy of patient results by comparing them with reference method results. These survey programs have been established in Korea using commutable reference materials with true target values traceable to CRMs and/or RMPs [13, 23, 24]. When inter-instrument non-comparability was detected, we obtained conversion factors traceable to reference target values using commutable reference materials that had been used for national accuracy-based surveys, instead of results from the reference instrument (AU5800). The inter-instrument CVs of converted results for creatinine and cholesterol were lower than those of LP, but higher than those of electrolytes.
A limitation of our weekly comparability verification was that we arbitrarily assigned the AU5800 instrument as the comparative reference system. Although AU5800 is the main instrument used in our laboratory with a closed system that uses exclusive calibrators and reagents for clinical tests with periodic monitoring of accuracy by performing PT from KEQAS and CAP, it is also susceptible to random or systematic errors, which could affect the comparability to the other instruments. However, we could detect errors in AU5800 indirectly since the other instruments showed collective bias to the reference instrument.
Another limitation of this study was that we could not guarantee the commutability of the samples for weekly comparability testing, even though we tried to pool serum samples without interfering materials. However, we used fully commutable materials for cholesterol and creatinine level measurements in the simplified comparison. The final limitation was that several comparability verification failure events were still observed owing to the fast, unexpected elevation of the PBIAS following changes in the reagent or calibrator. To prevent verification failure events, it is necessary to carefully check comparability results when there are specific events that can greatly affect within-laboratory harmonization. Our simple comparability verification is not a complete alternative to the CLSI EP31-A-IR guidelines [5], but an adjunct to improve intra-laboratory harmonization and minimize bias.
In conclusion, we succeeded in maintaining within-laboratory comparability of clinical chemistry tests from multiple instruments for five years employing a more frequent (weekly) and less laborious periodic comparability verification method using pooled residual human sera. Our method may assist the existing labor-intensive and complicated method based on the CLSI EP31-A-IR guidelines [5]. This comparability verification method can be applied to large healthcare centers or clinical laboratories that use multiple types of instruments.
XML Download