PDF
ePub
Citation
Print
INTRODUCTION
The diagnosis and management of pulmonary nodules can vary widely based on their probability of malignancy and characteristics (1). For indeterminate pulmonary nodules, typically with low-to-intermediate probability of malignancy, image-guided biopsy, excisional biopsy, or computed tomography (CT) surveillance is recommended (2). Percutaneous transthoracic needle biopsies (PTNBs) are often the first option for patients with peripheral pulmonary nodules (3).
Image-guided PTNB is an established diagnostic procedure with extensive evidence of its efficacy and safety when performed by qualified physicians in an appropriate environment (4). Cone-beam CT (CBCT)-guided PTNB with a virtual navigation system has shown high diagnostic performance with accurate and safe needle displacement, even for small pulmonary nodules (567). However, to achieve acceptable performance, including a high diagnostic success rate and low complication rate, each operator must have sufficient training and experience to develop competent procedural skills (4). Unfortunately, thus far, no guidelines have been published on how to assess trainees' performance in PTNB procedures and how much experience trainees should obtain to gain competence and be considered highly experienced operators.
The cumulative summation (CUSUM) method, a tool for evaluating the learning curve of a specific procedure (8), has been widely adopted as a critical tool in the quality control processes of the manufacturing and energy industries. The basic concept of a learning curve is that performance improves with time and experience, resulting in increased productivity (9). Recently, CUSUM has also garnered much attention in the medical field for the assessment of learning curves and as a quality control method (10), particularly in the fields of surgery and anesthesia (111213). It has been applied to lung nodule biopsy, and specifically for bronchoscopy-guided aspiration or biopsy (14151617); however, thus far, no studies have assessed the learning curve of PTNBs. Therefore, the purpose of this study was to evaluate the learning curve for CBCT virtual navigation-guided PTNBs using the CUSUM method and to determine the amount of experience needed to develop the appropriate skills for this procedure.
MATERIALS AND METHODS
This retrospective study was approved by the Institutional Review Board of Seoul National University Hospital and the requirement for informed patient consent was waived.
Patients
From March 2011 to December 2014, 2158 consecutive CBCT virtual navigation-guided PTNBs were performed by 7 board-certified radiologists who had started their fellowship training without any prior experience in CBCT virtual navigation-guided PTNBs. All operators attended a lecture on PTNB procedures and observed and practiced the procedure under the supervision of an experienced radiologist. The patients' medical records, CT findings prior to PTNBs, procedural records, post-PTNB follow-up studies, and diagnosis of biopsied lesions were retrospectively reviewed by one author.
Outcome Measures
In this study, the learning period until acceptable diagnostic performance and pneumothorax occurrence were achieved was investigated because these are thought to be the main quality indicators of the accuracy and safety of PTNB procedures. The final diagnosis of each biopsied lesion was determined based on biopsy specimens, surgical specimens, or follow-up images (618): 1) If the patient underwent surgical resection, the pathologic report of the surgical specimen was used to confirm the final diagnosis. 2) If the pathologic report of the biopsy specimen revealed a definitive malignant or specific benign pathology, such as tuberculosis or hamartoma, they were accepted as the final diagnosis. 3) In the case of nonspecific benign pathologies, lesions were determined to be benign only if they decreased by 20% or more in diameter over the follow-up period or remained stable in size for more than 2 years. 4) If lesions with nonspecific benign pathology did not show a sufficient interval decrease in size or were followed up for less than 2 years, they were considered to be indeterminate. Pneumothorax was recorded based on post-procedural CBCT and follow-up chest radiographs after PTNBs.
Construction of the CUSUM Chart
The standard and risk-adjusted CUSUM (RA-CUSUM) methods for calculating the diagnostic performance and pneumothorax occurrence were as follows (19). With respect to the outcome, accurate diagnosis (true positives and true negatives) and the absence of pneumothorax were considered as cases of “success.” Technical failures and diagnostic errors (false positives and false negatives) and the occurrence of pneumothorax were considered as cases of “failure.”
The score (s) was recorded from p0, indicating an acceptable failure rate (the level of inherent error if the procedure is performed correctly), and p1, indicating an unacceptable failure rate. The CUSUM score was generated at the beginning as 0. After each case of success, an amount s is subtracted from the previous CUSUM score, and an amount 1-s is added to the previous CUSUM score for each case of failure. A successful outcome leads to downward movement of the CUSUM score, whereas a failed outcome leads to upward movement of the CUSUM score. When the CUSUM lines crossed a predetermined lower decision limit (H0) from above, the failure rate was considered to be within the preset acceptable rate. When the CUSUM lines crossed the upper decision limit (H1) from below, the failure rate was considered to be unacceptable. When the CUSUM lines remained between the upper and lower decision limits, whether the performance was acceptable or unacceptable was considered inconclusive and continued observations were required. These limits were calculated based on the risk for type I and type II errors. A type I error (α) is the probability of failing to cross the acceptable limit when the true failure rate is within the acceptable range, whereas a type II error (β) is the probability of downward crossing of the acceptable failure rate limit when the true failure rate is not in the acceptable range. In this study, type I and type II error levels of 0.10 were used for all analyses. The p0 and p1 in this analysis were determined to be 0.06 and 0.12, respectively, for diagnostic accuracy, and 0.25 and 0.45, respectively, for pneumothorax occurrence. We set the acceptable failure rate at 0.06 based on the calculated failure rate reported in a recent meta-analysis (20) and the unacceptable failure rate was set at 0.12 (double the acceptable failure rate) (19). From among the wide range of reported pneumothorax rates of between 0.10 and 0.29, an acceptable pneumothorax occurrence rate of 0.25 was set (2021), with an unacceptable pneumothorax occurrence rate of 0.45 according to the suggested quality improvement threshold (22). Other values adopted in this study are summarized in Table 1.
Because standard CUSUM does not take the weight of the CUSUM score into account depending on the degree of difficulty expected in the procedures, a RA-CUSUM was constructed. Rather than assuming that the acceptable failure rate is the same for all patients, the risk of failure was predicted to vary because of the difficulty of the procedure depending on the characteristics of the patient and lesion (620). In the RA-CUSUM, the score si was defined by the predicted risk of failure (pi) of each procedure, which was calculated using a logistic regression model based on baseline variables that have been reported as factors associated with diagnostic performance and pneumothorax occurrence (23). The predicted risk of diagnostic failure was calculated using a logistic regression model of 2 variables: lesion size (≤ 1 cm vs. > 1 cm) and lesion location (lower vs. upper/middle) according to a study by Lee et al. (6). The predicted risk of pneumothorax was calculated using a logistic regression model of the available variables in this study among the documented risk factors (20): age, sex, lesion size, lesion location, emphysema, and number of pleural passages. Thereafter, the value of si was defined as follows, where the odds ratio (OR) in this study was calculated using p0 and p1 for easy comparison with the standard CUSUM.
Statistical Analysis
The CUSUM chart was constructed using Microsoft Excel 2013 (Microsoft Corporation, Redmond, WA, USA), and the difference between the proportions of patients' clinical features and target lesions' characteristics among the operators was evaluated using Pearson's χ2 test. All statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS version 23.0, IBM Corp., Armonk, NY, USA), with a p value of < 0.05 considered to indicate a statistically significant difference.
RESULTS
Operators 1–7 performed 286, 319, 601, 93, 292, 116, and 335 procedures, respectively. The number of completed procedures depended on the duty months for this procedure. The characteristics of the patients and lesions for each operator are shown in Table 2. Their overall diagnostic success rates and pneumothorax occurrence rates are described in Table 3.
Diagnostic Performance
Of the 2158 PTNBs, 6 cases were excluded because of the lack of pathologic results (n = 3) or because the targets were non-pulmonary targets (n = 3). In addition, 110 cases were excluded because their final diagnoses were not determined. Therefore, the diagnostic accuracy of CBCT virtual navigation-guided PTNBs in this study was calculated based on 2042 PTNB procedures in 1948 patients (1147 males and 801 females; mean age, 63 ± 12 years). The 2042 CBCT virtual navigation-guided PTNBs included 123 diagnostic failures (6.0%): 9 false positives, 74 false negatives, 5 technical failures, and 35 non-diagnostic PTNBs (e.g., “insufficient for diagnosis,” “only necrotic materials”). The remaining 1919 cases were correctly diagnosed and defined as cases of diagnostic success, of which 1402 were true positives and 517 were true negatives.
A wide range of time to acquire appropriate skills for diagnostic performance was observed. Figure 1 shows the standard and RA-CUSUM scores for the diagnostic performance of each operator. Six of the 7 operators (85.7%) attained an acceptable failure rate of 6% in diagnostic performance after a median number of 105 PTNB procedures (95% confidence interval [CI], 14–240). The CUSUM chart of Operator 6 remained between the decision limits even after 116 procedures, indicating the requirement for further observation.
The RA-CUSUM plots showed a similar pattern to that of the standard CUSUM plots, and 5 (71.4%) of the 7 operators achieved an acceptable failure rate after a median number of 93 PTNB procedures (95% CI, 39–142). The RA-CUSUM chart demonstrated that Operators 6 and 7 did not reach H0 and remained between the decision limits.
Pneumothorax Occurrence
The learning curves for pneumothorax occurrence are shown in Figure 2. For 6 of the 7 operators, the CUSUM line for pneumothorax occurrence demonstrated a downward trend and crossed the decision limits after a median number of 79 PTNBs (95% CI, 27–155) on the standard CUSUM chart. On the RA-CUSUM chart, all operators were shown to have attained competence after a median number of 80 PTNB procedures (95% CI, 38–127).
DISCUSSION
In this study, we assessed the learning curve of image-guided PTNBs and demonstrated the utility of the CUSUM method of monitoring operator performance, particularly in CBCT virtual navigation-guided PTNB. The CUSUM method is a relatively simple statistical method that can assess the competence of operators and monitor performance in the medical field (19). It can demonstrate how individuals acquire proficiency over time and maintain competence. According to our study results, the diagnostic failure and pneumothorax occurrence rates decreased with the increasing experience of the operators. However, a wide range of experience was found to be necessary to attain an acceptable failure rate for diagnostic performance. Operators 1 and 5 acquired the earliest competence after 44 procedures, whereas Operator 7 achieved the latest competence after 333 procedures. Regarding pneumothorax occurrence, the standard CUSUM analysis revealed that the complication rate after a median of 79 procedures was similar or decreased for all but Operator 4.
The amount of experience to achieve proficiency might vary depending on the trainee and the level of performance required (24). This study indicated high variability in the learning process for CBCT-guided PTNB, ranging from approximately 40 cases to more than 300 cases. Several factors may explain this finding. One possible reason is that a different proportion of variables could have affected the outcomes, despite adjustment for those factors. Differences in the prevalence of malignancy, which also determined whether each operator performed a procedure, may also have affected diagnostic performance. In addition, the CUSUM results depended on the order of success and failure, as well as overall performance, such as the diagnostic accuracy or pneumothorax occurrence rate. Alternatively, 2 operators in our study may have been outliers who need more experience to become proficient. High variability in the amount of experience required to learn a skill is not uncommon and has been reported in several studies on medical procedures (121524). Thus, individualized training under expert supervision may be necessary.
Several issues must be considered when CUSUM analysis is used in clinical practice; specifically, CUSUM analysis requires the specification of multiple values for outcome measurements. Determining acceptable and unacceptable failure rates is essential, and an acceptable failure rate should be the most accurate estimate of the failure rate of a skilled operator (19). In our study, we set the acceptable failure rate at 0.06 based on the calculated failure rate reported in a recent meta-analysis of CBCT-guided PTNB (20); however, a failure rate of 0.06 can be considered low because the pooled mean success rate of all types of image-guided PTNB is 0.89 (22). The high diagnostic accuracy of CBCT-guided PTNB compared with other modalities in the literature may have been caused by an increase in accuracy resulting from the advantages of the CBCT system or from publication bias in the initial period of this new modality. In addition, diagnostic failure in our study included “nondiagnostic” or technical failure, as well as false positives and false negatives, and therefore a difference inevitably exists between diagnostic success and diagnostic accuracy. We then simulated how the acceptable and unacceptable failure rates would affect the outcome of the CUSUM plots in diagnostic performance (Fig. 3). If a more generous acceptable failure rate of 0.10 were adopted, all operators would have achieved proficiency after a median of 33 procedures. Conversely, if a harsher failure rate of 0.04 were adopted as the acceptable failure rate, only 2 operators would have been able to reach the lower decision limit, whereas 2 operators would have crossed the upper decision limit, suggesting poor performance, even after finishing their training.
Knowledge of the learning curve may be particularly useful in a medical training program. In general practice, trainees perform medical procedures for a certain period of time under the supervision of specialists and then, after attaining proficiency, perform them independently. However, difficulties exist in judging whether trainees have truly achieved competency. The CUSUM method assures trainees and supervisors that learning is proceeding favorably if the graph goes downward to a lower decision limit and determines whether the operator is becoming more competent. In addition, CUSUM may also be useful for quality control, such as monitoring an operator's ability and identifying poor performance (19). Our study demonstrated this potential in the pneumothorax occurrence chart. The plot of Operator 2 was above the upper limit after 10 procedures, which indicated a higher failure rate than the unacceptable failure rate of 0.45. However, the plot moved down after more procedures were performed, eventually crossing the lower decision limit after 102 PTNBs, and competence was maintained thereafter. Thus, if the CUSUM method is applied in real clinical practice, it can help identify deteriorated performance, thereby providing an opportunity to intervene and ensure the operator's improvement before too many adverse events can occur.
Our study had several limitations. First, setting different failure rates may lead to substantially different outcomes regardless of whether the trainee is proficient. Therefore, careful interpretation of the CUSUM chart is warranted. Nevertheless, we believe that CUSUM analysis can be a useful and intuitive method of evaluating an operator's performance, which can determine the learning curves necessary for different radiology interventional procedures (CBCT virtual navigation-guided PTNBs in this case). Second, for the general application of CUSUM in assessing the learning curve and monitoring quality control, a guideline for setting values should be provided by an authorized committee, which warrants further study with a larger population.
In conclusion, operators' performance, including diagnostic accuracy and pneumothorax occurrence, in CBCT virtual navigation-guided PTNBs was demonstrated to improve with increasing experience over a wide range of learning periods.
XML Download