This retrospective multicenter study was approved by the Institutional Review Boards of all five participating institutions, which are academic tertiary hospitals in Korea. The requirement for written informed consent for the retrospective review of medical records and image data was waived.

Investigators from five institutions searched the databases to identify patients aged 18 years or older who met the following inclusion criteria: 1) at risk of developing HCC (hepatitis B virus [HBV] carrier, hepatitis C virus carrier with chronic hepatitis, or cirrhosis from any causes), 2) pathologic diagnosis of focal hepatic lesions based on surgery or biopsy findings between January 2008 and July 2014, 3) both dynamic CT and gadoxetate-enhanced MRI performed within 2 months before the pathologic diagnosis, and 4) no past history of hepatic malignancy. Of the 2958 patients who met the inclusion criteria, 287 were excluded because they had undergone local ablative treatments (n = 274), had ruptured HCCs (n = 3), had active extrahepatic malignancy (n = 6), or had false-negative biopsy results (n = 4). From the remaining 2671 patients, 103 patients with 117 non-HCC lesions (59 benign lesions and 58 non-HCC malignancies) were included; we also randomly selected 114 patients with 114 HCCs. Our final study population consisted of 217 patients with 231 pathologically-confirmed hepatic lesions (Fig. 1).

MRI was performed using 1.5T or 3T scanners. The protocols included a dual-echo T1-weighted gradient-recalled echo sequence, moderately and heavily T2-weighted turbo spin-echo sequences, dynamic three-dimensional T1-weighted gradient-recalled echo sequence using gadoxetic acid disodium (Primovist, Bayer AG, Berlin, Germany) as the contrast medium, and diffusion-weighted imaging. For dynamic imaging, arterial-, portal venous-, transitional-, and hepatobiliary-phase images were acquired 25–35 seconds, 60–70 seconds, 2 or 3 minutes, and 15 or 20 minutes after contrast injection, respectively. Contrast-enhanced dynamic CT was performed using 16-, 40-, or 64-detector scanners. After acquiring unenhanced images, arterial-, portal venous-, and delayed-phase images were obtained approximately 30 seconds, 70 seconds or 90 seconds, and 180 seconds or 210 seconds after iodinated contrast medium injection, respectively. Technical details for CT and MRI acquisition are provided in Supplementary Material 1.

Clinical, pathologic, and imaging data were retrieved from the database of each participating institution, which included information about patient demographics, the cause of chronic liver disease, serum alpha-fetoprotein level, child-Pugh class, pathologic diagnosis, and CT/MR images. They were then sent to the central site (Severance Hospital), where the image data were anonymized, randomized, and assigned new identification numbers. The processed imaging data, along with screen-captured images marking the lesions' sites (which was performed by an abdominal radiologist with 4 years of experience), were then sent back for image analysis. The data collection and preparation steps are described in detail in Supplementary Material 2.

Four board-certified abdominal radiologists from different institutions (Reviewer 1, Reviewer 2, Reviewer 3, and Reviewer 4, with 5, 6, 10, and 21 years of experience, respectively, in abdominal CT and MRI) independently analyzed the CT/MR images and submitted the results to the central site. In the first round, they analyzed the MR images to evaluate the LI-RADS features and determine the initial LI-RADS category in each case without assessing the ancillary features. Then, they evaluated the ancillary features and assigned the final LI-RADS category. They did not evaluate the presence or absence of threshold growth. In the second round conducted at least 8 weeks after the first round to minimize recall bias, they evaluated the CT images in the same manner. In cases where a hepatic lesion could not be delineated in the marked area, it was recorded as “not visible” and considered benign. The washout appearance was determined twice, first as defined in the LI-RADS by using the portal venous phase alone (111314), and then by using both portal- and transitional-phase images. The reviewers were asked to characterize a feature as absent if they could not unequivocally determine whether the feature was present or absent due to suboptimal or poor image quality.

A detailed description of the image analysis protocol is provided in Supplementary Material 3.

For sample size estimation, the expected per-lesion sensitivities of dynamic CT and MRI for HCC were assumed to be 68% and 80%, respectively, with the same specificity of 94% (19). On the basis of these expected values and a power of 80%, we calculated that we would need 209 or more hepatic lesions to obtain a significant difference in diagnostic performance with a two-sided type I error of 5% (20).

The baseline characteristics were compared using analysis of variance for continuous variables and the chi-squared test for categorical variables. Multiple comparisons were corrected using Bonferroni's method. The frequency of image features was compared using chi-squared or Fisher's exact tests. We used a generalized estimating equation method to compare the diagnostic performance between dynamic CT and MRI for distinguishing hepatic malignancies from benign lesions categorized as LR-5, LR-5V, or LR-M; for differentiating HCC from non-HCC lesions categorized as LR-5 or LR-5V; for differentiating non-HCC malignancies from other lesions categorized as LR-M; and then in subgroups containing only two disease entities for differentiating between HCC and benign lesions (with other malignancies excluded) and between HCC and other malignancies (with benign lesions excluded). In addition, we examined the diagnostic performance of LR-5 (without LR-5V) for HCC. We used LR-5 alone, not including LR-4, for the calculation of diagnostic accuracy, because LR-5 alone is considered to indicate “definite HCC” in LI-RADS (111314). Lastly, we calculated the diagnostic performance of gadoxetate-enhanced MRI when using the portal phase alone for washout evaluation and compared it with the results obtained using both the portal and transitional phases. Pathologic diagnosis was used as a reference standard. To examine the added value of the ancillary features, we tabulated the LI-RADS categories before and after applying the ancillary features and calculated the net reclassification improvement. The net reclassification improvement is a statistic for assessing the improvement in performance gained by adding a new factor to a model by measuring the extent to which individual subjects with and without disease are appropriately reclassified into more appropriate categories (21). We performed this analysis only on MRI cases after excluding the LR-M and “not visible” cases. Kappa statistics were computed as indices of inter-reader agreements between the four readers. Kappa statistics were used to indicate agreement, with 0.8–1.0 indicating excellent agreement; 0.60–0.79, good agreement; 0.40–0.59, moderate agreement; 0.20–0.39, fair agreement; and 0–0.19, poor agreement. All statistical analyses were performed using SAS 9.2 (SAS Institute Inc., Cary, NC, USA) and R version 3.3.3 (The R Foundation for Statistical Computing, Vienna, Austria). P values < 0.05 were considered statistically significant. A detailed description of our statistical analysis is provided in Supplementary Material 4.

The baseline characteristics of the 217 patients with 231 hepatic lesions (114 HCCs, 58 other malignancies, and 59 benign lesions) are summarized in Table 1. Patients with other malignancies were significantly older compared to those with HCCs (p = 0.017) and benign lesions (p = 0.006). HBV was a more common etiology of chronic liver disease in patients with HCCs than in those with benign lesions (p = 0.003). Tumor size was the largest in patients with other malignancies and the smallest in patients with benign lesions (p < 0.001), and biopsy was more frequently used for the diagnosis of other malignancies or benign lesions than HCCs (p < 0.001). The interval between CT and MRI ranged from 0 to 64 days, with a median interval of 12 days. Pathologic diagnosis was performed within 2 months after the first imaging study as it was one of our inclusion criteria.

The frequencies of HCCs, other malignancies, and benign lesions according to the LI-RADS category and imaging modality are presented in Figure 2 (only pooled results) and Supplementary Table 1 (including full results). Minor proportions of non-HCC malignancies were categorized as LR-5V (3.4% [8/232] for CT and 4.7% [11/232] for MRI) (Figs. 2, 3).

The frequencies of major features—arterial-phase hyperenhancement, washout appearance, and capsule appearance—according to imaging modality are presented in Supplementary Tables 2 (by disease) and 3 (by size, only including HCCs). There were seven cases in which at least one reviewer decided that evaluation of arterial-phase MR images was suboptimal because of severe motion artifacts or too early acquisition. In HCC lesions, diffuse arterial hyperenhancement and washout appearance either in the portal or transitional phase were more frequently observed on MRI than on CT (88.2% vs. 82.5% [p = 0.015] and 90.4% vs. 77.2% [p = 0.005], respectively), but the frequency of washout appearance observed on MRI became lower than that on CT when the washout appearance was determined in the portal phase alone (71.1% vs. 77.2% [p = 0.034]). Rim-like arterial-phase hyperenhancement, a feature favoring malignancies other than HCC, was more frequently observed on MRI than on CT in non-HCC malignancies (44.8% vs. 32.3%, p = 0.006), but not in HCC (4.6% vs. 6.8%, p = 0.205) or benign lesions (7.2% vs. 5.5%, p = 0.572).

Analysis of the pooled data revealed that the ancillary features modified the final LI-RADS category in 110 (12.0%) of 918 cases (Supplementary Table 7). None of the LR-4 cases was upgraded to LR-5, as it is not warranted by the LI-RADS (7). Of the 686 malignant cases, 22 (3.2%) were correctly upgraded into higher LI-RADS categories, while 25 (3.6%) were incorrectly downgraded into lower categories. Of the 232 benign cases, 36 (15.5%) and 28 (12.1%) were reclassified into higher and lower LI-RADS categories, respectively. The overall net reclassification improvement was estimated to be −0.055 (p = 0.117), indicating a negative impact on the overall categorization without statistical significance. However, of 32 and 19 benign cases initially categorized as LR-5/5V and LR-M, 10 (31.3%) and 4 (21.1%), respectively, were correctly downgraded into lower categories after applying ancillary features (Fig. 5).

When the LI-RADS categories were grouped into three categories of LR-5/5V, LR-M, and LR-1/2/3/4, inter-reader agreements were good for both CT and MRI (κ = 0.626 and 0.601, respectively). Inter-reader agreements were good for arterial hyperenhancement assessments using CT (κ = 0.675) and moderate MRI (κ = 0.563), for assessments of washout appearance by using CT and MRI (κ = 0.510 and 0.532, respectively), as well as for assessments of capsule appearance by using CT (κ = 0.476). However, inter-reader agreement was fair for assessments of capsule appearance by using MRI (κ = 0.326).