PDF
Citation
Print
INTRODUCTION
Internal quality control (QC) plays a significant role in a clinical laboratory. Conventional QC procedures rely on the periodic measurement of commercial QC materials with a known concentration prepared in matrices; however, these artificial serum matrices may substantially differ from patient-derived serum samples [1]. Despite the numerous QC procedures for guiding laboratories to determine the most appropriate QC testing frequency [2], in China, many laboratories analyze only two concentrations of QC materials once a day [3].This frequency may be insufficient to rapidly detect analytical errors [4].
In response to these shortcomings,another QC procedure was proposed based on patient results, which can provide real-time error monitoring [5–12]. This patient-based real-time QC (PBRTQC) is performed by monitoring patient population parameters, such as the mean, median, or standard deviation (SD) of analyte values [11,13,14]. PBRTQC does not require the use of QC materials and testing, and instead relies only on calculations using existing patient results, thus avoiding commutability problems and minimizing cost. Fleming, et al. [15] Reported that PBRTQC could decrease the utilization of conventional QC materials by about 75–85%. Among the several types of PBRTQC procedures, the moving average (MA) was the first and has been used most widely. However, the MA procedure cannot detect all types of biases for a broad range of analyte distributions. For example, van Rossum and van den Broek [12] and Liu, et al. [16] suggested that the MA procedure cannot rapidly detect a small positive bias for prostate-specific antigen and carcinoembryonic antigen assays, respectively.
In 2015, the European Society of Cardiology guidelines produced rule-in and rule-out algorithms using high-sensitivity cardiac troponin T (hs-cTnT) assays in patients with suspected non-ST segment elevation myocardial infarction [17]. For such assays, even a small shift can lead to quite different interpretations of results and consequent patient management, thereby requiring further invasive investigations and corrective actions. Therefore, new QC procedures other than conventional QC or MA QC proceduresare needed to detect a small critical bias for highly sensitive assays such as the Elecsys hs-cTnT assay (Roche Diagnostics, Mannheim, Germany), which is used in most laboratories. Liu, et al. [16] first demonstrated that the moving rate (MR) of prostate-specific antigenresults larger than 30 ng/L as a QC tool could be used to rapidly detect a positive bias of 30 ng/L, which went undetected using conventional QC procedures. However, they did not study whether MR QC procedure was appropriate to detect a larger bias, and the performance of MR QC procedure has not been validated.
We explored whether conventional QC procedures, the MA procedure, or a new PBRTQC procedure based on the MR of positive patient results could be used to rapidly detect a small critical bias (e.g., 5 ng/L) for the hs-cTnT assay. This is the first study to comprehensively explore the ability of the MR procedure to detect biases of different sizes and to use bias detection curves and validation graphs for comparing MR procedures with different parameters and validate the performance of the optimal MR procedure.
MATERIALS AND METHODS
Patient data and hs-cTnT assay
We retrospectively reviewed 37,047 hs-cTnT assay results during eight months (January 1 to August 30, 2019) after excluding the results from 56 research samples, five proficiency samples, and 496 internal QC samples from the laboratory information system (LIS) of the Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Guangzhou, China. We then organized the hs-cTnT assay results according to the reporting time to the LIS. hs-cTnT concentrations were measured from patient plasma samples in our hospital using Elecsys on the Roche Cobase 602 analyzer (Roche Diagnostics). There were three out-of-control points, which triggered the 13s rule (a run is rejected when a single control measurement exceeds the mean±3 SD control limit) over the eight months. One of the out-of-control points was due to the degradation of subpackaged QC material; to resolve this issue, we tested the new redissolved QC material. The other two out-of-control points were caused by bubble formation in controls during sample preparation; therefore, we retested the controls and retrained the new employee to avoid this issue. There was no out-of-control point caused by the hs-cTnT analyzer, method, or reagent in the internal QC chart using the 13s/22s/R4s multi-rule procedure [18] in our laboratory, and the clinicians did not question the hs-cTnT concentrations obtained. The hs-cTnT concentrations exhibited skewed distribution. All data were analyzed using Microsoft Excel 2007 (Microsoft, Redmond, Washington, USA). Since fewer samples were obtained on weekends and holidays, the daily run size of hs-cTnT assay was calculated only for working days. Overall, 29,989 patient results were obtained on 167 working days from January to August, resulting in a daily run size of 180 patient results.
Use of conventional QC procedures to detect a critical bias
The commercial QC materials were produced by Bio-Rad Laboratories (Hercules, CA, USA). In our laboratory, two concentrations of QC materials were analyzed once a day. The means of eight months of QC results for the two concentrations of QC materials were 254 ng/L and 995 ng/L, and the analytical coefficients of variation (CVa, calculated as SD/mean) were 3.9% and 4.0%, respectively. The ability of conventional QC procedure to detect a 5 ng/L bias at these two concentrations was then examined using the13s/22s/R4s multi-rule procedure as deviation rules.
To obtain the probability of a positive bias of 5 ng/L triggering one QC rule, the standard z-value was first calculated. First, we assumed that the QC concentrations follow a Gaussian distribution. Then, the standard z-value for the probability of obtaining a QC concentration larger than the N SD (N=1, 2, or 3) in the presence of a critical bias was calculated as follows [16]:
where Meanold and Meannew denote the mean concentration of a specific QC result before and after introducing critical bias, respectively.
Next, the probability of obtaining a QC concentration smaller than N SD can be obtained by referring to the z-table, which is denoted as p; accordingly, the probability of obtaining a concentration larger than N SD under the critical bias is 1–p, and the probability of obtaining two consecutive QC concentrations exceeding N SD (i.e., the 2NS rule) is calculated as (1–p)2. Similarly, the probability of obtaining four consecutive QC concentrations exceeding N SD (i.e., the 4NS rule) is calculated as (1–p)4. We did not calculate the probability of triggering the R4s rule (a run is rejected when 1 control measurement in a group exceeds the mean plus 2 SD and another exceeds the mean minus 2 SD within a run), which is mainly used to monitor random error.
Finally, we would select an appropriate low QC concentration (denoted as Meanlow) that could detect a small critical bias of 5 ng/L. For example, if the desired power of detection for critical bias is set to 95% and the QC rule is 12s, then the low QC concentration could be calculated as:
Rearranging the above formula, we obtain:
where z=−1.65 for a 95% power of detection.
As an example, we illustrate the method of obtaining the probability of triggering 41S for a low concentration of 254 ng/L. First, we calculate the standard z-value as follows:
Based on the z-table, the p-value is 0.6879; therefore, the probability of triggering the 11S rule is 1–p=0.3121, and the probability of triggering the 41S rule is 0.31214=0.0014.
To choose the appropriate low QC concentration for detecting the positive bias of 5 ng/L, we assumed that the CVa was the smaller of 3.9% and 4.0% (i.e., 3.9%). Hence, the low QC concentration was calculated as:
Determination of MA QC procedure parameters
The MA QC procedure settings can be divided into three parts: (1) inclusion and exclusion criteria by applying truncation limits to exclude all values that are above or below a certain threshold; (2) MA calculation method, including the MA calculation algorithm and the block size,defined as the number of patient results to average in the algorithm; and (3) control limits.
When there were no truncation limits in the MA QC procedure, “extreme” patient results (e.g., critical values) that differed substantially from the mean analyte concentration could not be excluded, which would lead to a higher false rejection rate [11, 13, 15]. In our preliminary analysis, where no truncation was used, the false rejection rate reached up to 23%. Since hs-cTnT concentrations higher than 150 ng/L are considered critical values in our laboratory, these results were truncated to improve the performance of the MA procedure. The false rejection rate was calculated as the proportion of unaffected results that fell outside the control limit. Approximately 19% of the raw data were truncated, which were considered excessive, and block sizes of 20, 50, 100, 200, 500, and 1,000 were investigated. MA was then calculated as:
where Z(i) denotes the calculated mean for hs-cTnT result i, and X(i) denotes the hs-cTnT result for sample i. The MA procedure was operated in continuous mode, so that a new MA value was calculated for every new hs-cTnT assay result. The control limits were set using the mean and SD of the MA as follows:
Determination of MR QC procedure parameters
The MR QC procedure settings can be also divided into three parts, like the MA procedure. However, because of the robustness of the MR QC procedure algorithm, no truncation limits are used in this QC procedure.
For the MR QC procedure, the hs-cTnT concentrations should be converted into a binary status (negative=hs-cTnT concentration below the cut-off value; positive=hs-cTnT concentration above the cut-off value). We chose medical decision points of hs-cTnT concentrations [17] (5, 14, and 52 ng/L) as cut-off values. After converting negative and positive results to 0 and 1, respectively, we calculated the MR of 1 (denoting positive results) in block sizes of 20, 50, 100, 200, 500, or 1,000 as follows:
where MR(i) denotes the calculated moving rate for hs-cTnT result i and T(i) denotes the converted binary value (1 or 0) of the hs-cTnT concentration for sample i. A new MR value was obtained for every new hs-cTnT assay result with the MR procedure operated continuously. The control limits were set using the mean and SD of the MR for positive results as follows:
MA or MR bias detection simulation and validation
MA or MR bias detection simulation and validation were performed as described by van Rossum and Kemperman [9,10]and van Rossum [19]. Bias detection simulation was conducted to investigate the performance of different MA or MR procedures by calculating the median number of patient samples affected until a bias introduced into the dataset was detected (MNPed). Bias detection was defined as an MA or MR value falling outside of control limits. Biases ranging from 1 ng/L to 150 ng/L were introduced at nine random positions and differed by at least 2,000 assay results. In addition, before bias introduction, all MA or MR procedures were run for 2,000 hs-cTnT raw results. The number of patient samples affected by the introduced bias was then calculated as the number of patient samples falling between the point of bias introduction and the first MA or MR value exceeding the control limits. Maximum, minimum, and median (i.e., MNPed) numbers of patient samples needed for bias detection were determined. After comparing the MNPed across different MA or MR procedures with different block sizes, we selected an optimal MA or MR procedure for validation.
Bias detection curves and validation graphs were plotted as described by van Rossum and Kemperman [9, 10], with the introduced error on the X-axis and the MNPed on the Y-axis. Validation graphs were generated by plotting MNPed as bars, with the introduced error on the X-axis and the number of results needed for bias detection on the Y-axis; the maximum and minimum number of patient samples affected until bias detection was plotted as error bars. The validation graph was used to reflect the performance of a specific QC procedure. For PBRTQC procedures, we expected the error to be detected between the daily scheduled internal QC measurements. Therefore, we compared the MNPed with the average daily run size.
RESULTS
Use of conventional QC procedures to detect a critical bias of 5 ng/L
The probabilities of triggering different QC procedures when a positive critical bias of 5 ng/L appeared are summarized in Table 1.
Determination of MA or MR QC Procedure Parameters
Parameters of the MA or MR procedures are presented in Table 2. We did not investigate QC procedures for cases with the CVa larger than 20% owing to the wide control limits. The false rejections of these PBRTQC procedures were all <1%, with some even lower than 0.1%.
MA or MR bias detection simulation
To compare bias detection for a small systematic error corresponding to 5 ng/L hs-cTnT across MA or MR procedures, we used the MNPed values (Table 3). The MNPed of MA procedures with a corresponding block size were much larger than those of MR procedures when using a cut-off value of 5 ng/L or 14 ng/L. In addition, the MNPed of MR procedures with a cut-off value of 52 ng/L were all larger than 2,000.
As shown in Fig. 1, the MNPed of various QC procedures were plotted against increasing error. Based on these plots,we chose a block size of 100 for the MAand MR procedures with cut-off values of 5 and 14 ng/L, respectively, and a block size of 200 for MR procedures with a cut-off value of 52 ng/L as the optimal QC procedures. A randomly selected MR procedure to detect the introduced bias for hs-cTnT concentration measurement is presented in Fig. 2, which illustrates how the MR QC procedure detects a systematic error.
MA or MR validation
As shown in Fig. 3, in four optimal PBRTQC procedures, the detection of negative and positive bias was not equal. For the MA procedure, the MNPed increased again as bias grew larger than −20 ng/L. However, for the MR procedure, MNPed decreased with the increase in the negative bias and became constant when the positive bias was greater than the cut-off value. Hence, in the three optimal MR procedures, MR with a cut-off value of 52 ng/L could obtain the smallest MNPed for positive bias, while MR with a cut-off value of 5 ng/L could detect a very small bias (such as 5 ng/L) faster.
When comparing the MNPed with the average daily run size, only a positive bias of ≥20 ng/L was consistently detected by the selected MA procedure. The possibility of detecting a bias of 10 ng/L within a given day was 50% (Fig. 3A). For the MR procedure with a cut-off value of 5 ng/L (Fig. 3B), a positive bias of ≥5 ng/L and a negative bias of ≥−10 ng/L were consistently detected. There was an approximately 50% chance of detecting a bias of −8, −7, and −6 ng/L within a day. For the MR procedure with a cut-off value of 14 ng/L, a positive bias of ≥7 ng/L and a negative bias of ≥−50 ng/L were consistently detected. There was an approximate 50% chance of detecting a bias ranging from −16 to −10 ng/L within a day (Fig. 3C). For the MR procedure with a cut-off value of 52 ng/L, only a positive bias of ≥40 ng/L was consistently detected (Fig. 3D).
DISCUSSION
We demonstrated that the MR procedure with different cut-off values enables the rapid detection of different critical biases with a small chance of false alarms. MR procedures with a small cut-off value outperformed MA and conventional QC procedures for the rapid detection of a small bias.
Based on these results, there was only a small possibility (<10%) of detecting a positive bias of 5 ng/L by conventional QC procedures. To detect such small biases at 95% probability, the concentration of the QC material should be lower than 32.5 ng/L, which is often lower than that of available commercial QC materials. Consequently, laboratories may need to prepare for their QC samples in-house to meet the low concentration required. This requires pooling of patient samples, necessitating significant logistic planning and evaluation of stability and homogeneity, which may be difficult for some laboratories [20]. Like our findings for hs-cTnT, Liu, et al. [16] reported that conventional QC procedure could not detect a positive critical bias of 0.03 μg/L for a prostate-specific antigen assay. Therefore, it is likely that other assays with similar sensitivity will be drastically affected by even a small shift in the result, which could alter clinical interpretation. Thus, there is a need to develop a new QC procedure to make up for this defect in conventional QC procedure.
Although PBRTQC procedure has not been widely used in clinical laboratories, slow but continuous improvements have been made in its understanding and methodology, especially the MA procedure, corresponding with efforts toward achieving continuous analytical quality assurance in the last 50 years [21]. Recently, Ng, et al. [13] described a method for optimizing the MA procedure using a simulated annealing algorithm, and van Rossum and Kemperman [9] reported the use of bias detection curves and MA validation charts to choose an optimal MA procedure. These methods enable a laboratory to design an MA procedure objectively. In this study, we chose bias detection curves to select the optimal MA settings and used MA validation charts to validate the selected MA procedure. The MNPed was large for the detection of small bias of 5 ng/L, indicating that the conventional MA procedure may not be suitable for detecting bias in analytes with a skewed distribution [22].
Although other PBRTQC procedures, such as the average of delta, moving median analysis, and the exponentially weighted MA [7, 14, 23, 24],were not investigated in this study, the developed MR procedure not only rapidly detected a small critical bias (5 ng/L) but also showed that the MNPeds for detecting larger bias were smaller than the average daily run size. These MR procedures employed existing patient results and were easy to implement. Since hs-cTnT concentrations were converted into a binary status, the MR procedure was sensitive to changes in concentration close to the cut-off value, while being robust against excessively large concentrations. Therefore, we considered it unnecessary to use optimized truncation limits, allowing the inclusion of all the results.
MR procedures with a small cut-off value showed superior detection of a small bias but required more results to detect a large bias. In contrast, MR procedures with a large cut-off value showed superior detection of a large bias but could not detect a very small critical bias. Preferably, QC procedures should be able to detect all critical biases. For this reason, MR procedures with several cut-off values may address the needs of a robust QC system, like Westgard’s multi-rules with different performances [18]. In this study, we chose only three medical decision points derived from clinical guidelines [17] as the cut-off values. In the future, we will design a method for selecting an appropriate cut-off value for the MR procedure to meet the requirement of patient safety assurance.
The main limitation of our study was that we implemented the MR procedure for hs-cTnT assay only and did not include other analytes that might have a different distribution. Another limitation of our strategy is that we could not exclude the possibility of undetected systematic errors in our dataset, which might decrease the sensitivity of the QC procedure. Ideally, the raw data in a dataset are obtained after maintenance and calibration of instruments to ensure that the analyzer is in a good state for providing reliable results. Since we used Excel software, we performed only nine bias detection simulations on the dataset. Therefore, the obtained results, including the maximum, minimum, and median concentrations, may not be accurate, especially for some results with larger error bars. Clearly, more accurate data would have been obtained with more simulations performed, which further motivates us to develop a software that can be used to design and optimize the MR procedure (e.g., determining the need for truncation limits).
Given the high complexity of testing environments in clinical laboratories, the use of a single QC procedure may not be enough to detect different types of errors. We suggest considering PBRTQC procedure when the performance of the conventional QC procedure is limited, which was also recommended by van Rossum and van den Broek [12]. MA-based QC procedure could be examined and optimized using the web-based application, MA Generator (https://www.huvaros.com/) [12, 25]. However, laboratories are limited to using Excel to determine MR QC parameters owing to a lack of specific software for this purpose. After integrating the parameters and calculation algorithm into LIS, we will obtain a QC chart like that shown in Fig. 2.
We showed that the MR procedure is superior to the MA procedure in detecting a small bias. Use of the MR procedure with a smaller cut-off value may detect a small bias more rapidly. Although MR procedure with a larger cut-off value may detect a large bias, it may not effectively detect a small bias. Hence, MR procedures can be designed with different cut-off values to detect different biases more rapidly.
XML Download