This prospective study was approved by our Institutional Review Board and written informed consent was obtained from all patients before they underwent US. Patients who visited the thyroid clinic of the radiology department of Asan Medical Center for the evaluation of thyroid nodules were recruited between November 2017 and September 2018. The inclusion criteria used to select patients were 1) underwent US-guided core needle biopsy (CNB) or fine needle aspiration (FNA) or 2) follow-up for a thyroid nodule with decisive diagnosis. Patients were excluded from the study population if they 1) had a thyroid nodule less than 1 cm or 2) were younger than 18 years of age.

A decisive diagnosis consisted of a malignant or benign diagnosis. A malignant diagnosis was made when malignancy was confirmed based on a surgical specimen or using CNB histology or FNA cytology. A diagnosis of a benign nodule was made when any one of the following criteria was met: 1) confirmation using a surgical specimen; 2) benign CNB histology or FNA cytology findings; or 3) US findings of benign nodule (simple cyst, predominantly cystic nodule with reverberating artifact, or nodule with a spongiform appearance) (8) with no change over at least 1 year.

US examinations were performed using an RS80A US system (Samsung Medison Co., Ltd., Seoul, Korea) equipped with L3-12A (linear high-frequency probe; frequency range, 3–12 MHz). Real-time CAD system software (S-Detect™ for Thyroid, Samsung Medison Co., Ltd.) was integrated into the US system. This real-time CAD software provides two points to indicate the top left and bottom right of a region of interest (ROI) box enclosing a thyroid nodule on the US system. Based on the given box, the software automatically calculates the contour of the mass to distinguish it from normal thyroid tissue (segmentation) and evaluates the US features of the mass, including size (maximum diameter in captured image), composition (solid, partially cystic, or cystic), shape (oval-to-round or irregular), orientation (parallel or non-parallel), margins (well-defined, ill-defined, or spiculated), echogenicity (hyperechoic/isoechoic or hypoechoic or marked hypoechoic), and spongiform nature. These US features are quantified into computerized values and are presented as features to describe the thyroid nodule. Consequently, the software displays a diagnosis as to whether the nodule is possibly benign or possibly malignant (Figs. 1, 2). One radiologist with 7 years of experience in performing thyroid US drew the ROI box enclosing a target thyroid nodule on transverse and longitudinal images and then evaluated the quality of the nodule segmentation. If the segmentation did not properly define the contours of the nodule, manual correction was made.

All images were reviewed using a local picture archiving and communication system (PACS) monitor and digital imaging and communications in medicine image-viewing software (PetaVision, Asan Medical Center, Seoul, Korea). The US images were independently analyzed by three radiologists with different levels of thyroid US experience: resident (1 month, 200 cases), fellow (4 years, 1000 cases), and staff (7 years, 80000 cases). Resident and fellow radiologists were categorized as less-experienced radiologists and the staff radiologist categorized as an experienced radiologist. None of the reviewers had any information regarding the patients' clinical histories, previous imaging results, or previous biopsy results.

Two sets of US images for each patient were arranged in the folders of our local PACS; one set included gray scale images only and the other set included gray scale images and CAD system images together. The gray scale images included in the analysis were a transverse and longitudinal image of the target nodule used as a reference in the CAD system. Two separate US image analysis sessions were performed. First, each radiologist reviewed the grayscale images only and evaluated the following features: composition (solid, partially cystic, or cystic), echogenicity (hyperechoic/isoechoic or hypoechoic or marked hypoechoic), shape (ovoid-to-round or irregular), orientation (parallel or non-parallel), margin (smooth, spiculated/microlobulated, or ill-defined), and calcification (none, microcalcification, macrocalcification, or rim calcification). Each radiologist concluded their diagnosis as to whether the nodule was benign or malignant according to the previous report by Moon et al. (8). Each radiologist then re-evaluated the same grayscale images while referring to the CAD system and made a subjective diagnostic decision based on the grayscale US and CAD. Additionally, the conjunctive combination was also analyzed, with a finding of “malignant” on either the grayscale US or CAD system being defined as malignant.

The primary end point of this study was the diagnostic accuracy of the CAD system for diagnosis of malignant thyroid nodules in comparison with radiologists with different levels of experience. The secondary end points included the diagnostic performance and added value of the CAD system for diagnosis of malignant thyroid nodules, and the added value of the CAD system for interobserver agreement of US features between the three radiologists.

This study was primarily designed as a non-inferiority study and the sample size was estimated to determine the non-inferiority of the CAD to the radiologists regarding the primary end point (diagnostic accuracy). Non-inferiority was defined as a diagnostic accuracy that was no more than 10 percentage points below the estimated diagnostic accuracy of the radiologist. The diagnostic accuracy of thyroid US for the assessment of thyroid nodules is approximately 78%. To obtain a statistical power of 80% with a one-sided P-level of 0.05, a sample size of 157 patients in each group was required (20). Therefore, to allow for study dropouts, 200 patients were enrolled in this study.

The diagnostic performance of the CAD system and each radiologist was evaluated by calculating the sensitivities, specificities, positive predictive values (PPVs), and negative predictive values (NPVs), and then comparing those using generalized estimating equations for matched data.

To assess the added value of the CAD system, the sensitivity, specificity, PPV, and NPV were compared between the results from the grayscale images and those from the grayscale images with CAD. Generalized estimating equations were used for the two comparisons to account for clustering from the same patient.

Finally, the extent of interobserver agreement (the multiple kappa value) between the three radiologists in terms of the descriptions of the US characteristics was determined for the nodule evaluations using only the grayscale images and those made using both the grayscale images and CAD system. The level of agreement for Cohen's kappa was defined as follows: < 0.20, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and > 0.80, good agreement. Statistical analyses were performed using SAS software version 9.4 (SAS Institute Inc., Cary, NC, USA).

Diagnostic performance was assessed using the 165 patients with thyroid nodules with a final diagnosis. The diagnostic accuracy was 88.48% (95% confidence interval [CI], 82.65–92.53) for the CAD system and 83.03% (95% CI, 76.52–88.02) for the resident and fellow radiologists, which leads us to conclude that, with a 10% non-inferiority margin, the CAD system shows non-inferiority compared with the resident and fellow radiologists (p = 0.001). However, the diagnostic accuracy of the CAD was inferior to that of the staff radiologist (95.76%, 95% CI = 91.37–97.96; p = 0.138) (Table 2).

The sensitivity and NPV of the CAD system were significantly higher than those of the resident and fellow radiologists were, but were not significantly different to those of the staff radiologist. The diagnostic accuracy, specificity, and PPV of the CAD system were significantly lower than those of the staff radiologist were, but were not significantly different to those of the resident and fellow radiologists (Table 3).

When the radiologists subjectively combined the grayscale images and CAD, the conclusion was changed in six cases reviewed by the resident, three cases reviewed by the fellow, and one case reviewed by the staff. The diagnostic accuracy, sensitivity, PPV, and NPV slightly improved for all three radiologists, although the differences did not reach statistical significance (Table 4). The specificity was also slightly improved for the resident but showed no change for the fellow and staff radiologists. For the conjunctive combination analysis of the CAD system and grayscale images, the sensitivity, and NPV of the resident and fellow were significantly improved, while the specificity and PPV decreased (Table 5). The decreased specificity and PPV related to the increased overall number of positive cases, which increased the number of false positive cases. However, for the staff radiologist, the conjunctive combination resulted in significantly decreased specificity, PPV, and diagnostic accuracy, without a significant improvement in sensitivity or NPV.

Table 6 shows a summary of the interobserver variability in the US characteristics defined by the three radiologists before and after application of the CAD system. With the exception of shape (kappa = 0.034), moderate agreement was seen for all characteristics (kappa = 0.473–0.634) and they showed no significant difference after the CAD system was applied. The kappa value for shape was very low because of the possibility of the agreement occurring by chance. The proportions of ovoid shapes were 100%, 94.9%, and 99% for the resident, fellow, and staff radiologists, respectively.