PDF
Citation
Print
INTRODUCTION
The Liver Imaging Reporting and Data System (LI-RADS), which was last updated in 2018, offers a comprehensive framework for standardizing the terminology, technique, interpretation, reporting, and data collection when performing liver imaging for patients with a high risk for hepatocellular carcinoma (HCC), such as those with cirrhosis or chronic hepatitis B virus infection.1 LI-RADS has gained recognition as the standard for imaging diagnosis of HCC, and has been increasingly adopted worldwide after recent endorsement by the American Association for the Study of Liver Diseases.1
In the computed tomography (CT)/magnetic resonance imaging (MRI) LI-RADS v2018, LI-RADS category M (LR-M) is defined as observations that are probably or definitely malignant but not specific for HCC, thereby preserving the specificity of HCC diagnosis without compromising the sensitivity for detection of malignancy.1 Most LR-M category lesions are non-HCC malignancies such as intrahepatic cholangiocarcinomas (CCAs) and combined hepatocellular-CCAs (cHCC-CCAs), whereas 28-29% of LR-M lesions are HCCs with atypical imaging features.2,3 LR-M criteria include five targetoid imaging features of rim arterial phase hyperenhancement (APHE), peripheral washout, delayed central enhancement (DCE), targetoid restriction, and targetoid transitional phase (TP)/hepatobiliary phase (HBP) appearance, and the three non-targetoid features of infiltrative appearance, marked diffusion restriction, and necrosis or severe ischemia.1 LR-M is an exclusion criterion, which means that any hepatic observation showing only one targetoid imaging feature can be categorized into LR-M, and therefore, assessing individual LR-M features and assigning LR-M category accurately and reproducibly is important in the context of HCC diagnosis.
A recent multicenter retrospective study reported that agreement was poor for the diagnostic decision of LR-M compared with not LR-M (intraclass correlation coefficient [ICC], 0.46).4 However, this study did not provide the agreement for assessments of individual LR-M features, and was therefore of limited value for exploring the causes of poor agreement in the assignment of LR-M. Additionally, although several previous studies reported inter-reader agreement for the assessment of individual LR-M features,5-13 they were subject to limitations such as a single-center study design, a small number of study subjects, and variable results among studies.
Therefore, we performed a systematic review and meta-analysis of inter-reader agreement in the assessment of individual LR-M imaging features in the CT/MRI LI-RADS v2018 and explored the causes of poor agreement in LR-M assignment.
METHODS
This study was conducted and reported according to the guidelines of the meta-analysis of observational studies in epidemiology14 and preferred reporting items for systematic reviews and meta-analyses (Supplementary Table 1),15 and was prospectively registered in PROSPERO (ID, CRD42024505011). Two abdominal radiologists, each with ≥10 years of experience in liver imaging and ≥5 years of experience in performing systematic reviews with meta-analysis, independently conducted the literature search, study selection, data extraction, and assessment of study quality.
Literature search strategy
Comprehensive searches were conducted across the MEDLINE, EMBASE, and Cochrane databases to identify original research articles that reported inter-reader agreement in the evaluation of LI-RADS v2018 LR-M imaging features on dynamic contrast-enhanced CT/MRI. The search query was designed to ensure a sensitive literature search and to minimize the risk of overlooking relevant articles. Key search terms included HCC, LI-RADS, CT, and MRI. Search terms are defined in detail in Supplementary Table 2. The start date of the search period was set to January 1, 2018, to encompass all original articles utilizing LI-RADS v2018, and updates were performed until December 6, 2023. The search was restricted to studies comprising human subjects and those published in English. To broaden the search further, the bibliographies of the selected articles were scrutinized for potentially relevant studies.
Inclusion and exclusion criteria
After removing duplicates, articles underwent a review for eligibility based on the following criteria: (1) population-patients with HCC risk factors defined by LI-RADS,1 (2) index test: multiphasic CT or MRI, (3) comparator: no specific requirements, (4) outcomes: inter-reader agreement of LR-M imaging features, including five targetoid features (rim APHE, peripheral washout, DCE, targetoid restriction, and targetoid TP/HBP appearance) and three non-targetoid features (infiltrative appearance, marked diffusion restriction, and necrosis or severe ischemia), and (5) study design: any type of study, including observational studies and clinical trials. The exclusion criteria were as follows: (1) animal studies, case reports, review articles, editorials, abstracts/conference proceedings, meta-analyses, and systematic reviews, (2) studies outside the scope of interest of this investigation, (3) studies lacking sufficient data for extracting inter-reader agreement (e.g., absence of standard variance or 95% confidence intervals [CI]), and (4) studies with overlapping patient cohorts. The initial screening involved titles and abstracts, followed by full-text reviews of the selected potentially eligible abstracts.
Data extraction
Each study’s data were extracted using a standardized form prepared using Microsoft Excel (Microsoft, Redmond, WA, USA). The extracted information included study characteristics such as first author, year of publication, and study design (prospective vs. retrospective), subject characteristics, including sample size, sex, age, underlying liver disease, number of hepatic observations, percentages of HCC and/or other malignancy among observations, and observation size, type of imaging modality (CT vs. MRI), MRI scanner field strength, and contrast agent used, details related to imaging interpretation, such as the number of readers, level of experience, and whether readers were blinded to the reference standard, statistical method for evaluating inter-reader agreement, and study outcomes of interest. Inter-reader agreement was determined using kappa values (κ) with 95% CI. When an article lacked sufficient data, the corresponding authors were contacted via email to request additional information or clarification.
Assessment of study quality
The guidelines for Reporting Reliability and Agreement Studies16 served as the basis for evaluating the quality of the selected articles. The risk of bias was appraised across seven domains: (1) description of the index test (MRI techniques and imaging sequences), (2) the study subjects (recruitment methods and demographic characteristics), (3) the readers (number and experience level), (4) the reading process (availability of clinical information and independent review), (5) blinding to reference standard, (6) statistical analysis, and (7) the actual numbers of subjects and observations. A high-quality score was assigned to each domain if it was sufficiently detailed with no apparent potential bias.
Statistical analysis
The pooled κ values and their 95% CIs for the assessment of eight LR-M features were calculated using the DerSimonian-Laird random-effects model with Knapp and Hartung adjustment,17 based on the κ and 95% CI reported in the individual studies. κ values were categorized according to Landis and Koch as follows: poor (<0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), or almost perfect agreement (0.81- 1.00).18 Heterogeneity was assessed using Cochran’s Q test (P<0.1 indicating high heterogeneity) and I2 statistic (I2 >50% indicating substantial heterogeneity).19 Publication bias was evaluated using a funnel plot with a rank test. Subgroup analyses were performed using Cohen’s κ as a statistical method.
Subgroup meta-regression analysis with the following covariates was performed to explore the causes of study heterogeneity: (1) proportion of liver cirrhosis (≥50% vs. <50%), (2) dominant etiology of underlying liver disease (hepatitis B virus vs. others), (3) proportion of other malignancy among hepatic observations (≥10% vs. <10%), (4) mean size of hepatic observations (≥3 cm vs. <3 cm), (5) MRI scanner field strength (3.0-T only vs. 1.5-T or both), (6) MRI contrast agent (hepatobiliary contrast agent [HBA] only vs. extracellular contrast agent or both), and (7) number of readers (≥3 vs. 2). The analyses were performed using R version 4.3.2 (The R Foundation for Statistical Computing, Vienna, Austria), with P<0.05 considered statistically significant.
RESULTS
Literature search
Among the 1,029 articles identified through the search, 627 were screened after removing duplicates. Subsequently, 364 articles were excluded based on their titles and abstracts (Fig. 1). After a thorough full-text review, 239 articles were excluded from analysis. Finally, 24 eligible articles reporting the inter-reader agreement of LR-M imaging features, encompassing a total of 25 datasets,5-13,20-34 were included. Among them, 22 utilized only MRI. The remaining two articles used either CT or MRI, with one presenting the CT and MRI results separately22 and the other presenting them together.6
Study characteristics
The characteristics of the included articles are summarized in Table 1. The 24 eligible studies included 4,789 patients with 5,163 hepatic observations. One study was prospective,20 whereas the other studies were retrospective. Three studies exclusively included patients with cirrhosis.5,9,32 The dominant etiology of liver disease was hepatitis B virus in all but two studies, of which alcoholic liver disease was the dominant etiology in one9 and hepatitis C virus in the other.6 Twelve studies reported proportions of other malignancy such as CCA or cHCC-CCA that exceeded 10% of total observations.5-9,11,12,20,21,25,30,34 Twelve studies utilized only 3.0-T MRI scanners.8,11,12,20,21,23-25,27,31,33,34 Eighteen studies employed only HBA including gadoxetate or gadobenate dimeglumine as the MRI contrast agent.5,7,9-13,20,21,23,26-29,31-34 In all studies, CT/MRI interpretation was conducted by multiple image readers, with three or more readers participating in two studies.11,22
Quality assessment
All the included studies demonstrated a low risk of bias in at least six of the seven domains evaluated (Supplementary Table 3). However, five studies were deemed to have a high risk of bias concerning the study subject domain, primarily due to the absence of demographic characteristics such as patient age or the proportion of individuals with liver cirrhosis.7,8,10,27,28 One study was identified as having a high risk of bias in the readers’ domain because the imaging analysis was conducted by inexperienced readers, specifically radiology residents.33 Additionally, one study did not provide information regarding the experience levels of the image readers.25 Furthermore, the availability of clinical information during the review process was unclear in two studies.12,21
Inter-reader agreement for individual features in the LR-M criteria
All 24 eligible articles reported inter-reader agreement for rim APHE.5-13,20-34 Of them, 20 studies reported inter-reader agreement for peripheral washout,5-8,10-13,20-25,27,30-34 20 for DCE,5-12,21-27,29-32,34 18 for targetoid restriction,5-7,9-12,21-25,27,30-34 19 for targetoid TP/HBP appearance,5-12,21,23-27,30-34 12 for infiltrative appearance,5,6,8,9,12,13,20,22,25-27,29 12 for marked diffusion restriction,5-7,9,10,13,24-27,29,30 and 15 for necrosis or severe ischemia.5-7,9,12,13,20,22,24-27,29,30,33
Table 2 and Fig. 2 summarize the inter-reader agreements for the LR-M features. For rim APHE, 24 studies with 5,163 observations reported κ values ranging from 0.39 to 0.94 (Fig. 2A); with a pooled κ of 0.72 (95% CI, 0.65-0.78), indicating substantial inter-reader agreement. For peripheral washout, 20 studies comprising 4,134 observations showed κ ranging from 0.05 to 0.87 (Fig. 2B); with a pooled κ of 0.52 (95% CI, 0.39-0.65), suggesting moderate inter-reader agreement. For DCE, 20 studies with 3,900 observations reported κ ranging from 0.09 to 0.92 (Fig. 2C); with a pooled κ of 0.60 (95% CI, 0.50-0.70), indicating moderate inter-reader agreement. For targetoid restriction, 18 studies with 3,849 observations presented κ ranging from 0.20 to 0.97 (Fig. 2D); with a pooled κ of 0.68 (95% CI, 0.57-0.78), indicating substantial inter-reader agreement. For targetoid TP/HBP appearance, 19 studies with 3,998 observations reported κ ranging from 0.33 to 0.98 (Fig. 2E); with a pooled κ of 0.74 (95% CI, 0.65-0.83), indicating substantial inter-reader agreement. For infiltrative appearance, 12 studies with 2,192 observations reported κ ranging from 0.24 to 0.92 (Fig. 2F); with a pooled κ of 0.64 (95% CI, 0.49-0.78), indicating substantial inter-reader agreement. For marked diffusion restriction, 12 studies with 2,291 observations reported κ ranging from 0.03 to 0.98 (Fig. 2G); with a pooled κ of 0.49 (95% CI, 0.30-0.68), indicating moderate inter-reader agreement. For necrosis or severe ischemia, 16 studies with 3,118 observations reported κ ranging from 0.07 to 0.91 (Fig. 2H); with a pooled κ of 0.61 (95% CI, 0.48-0.73), indicating substantial inter-reader agreement. Additionally, the proportion of studies showing substantial to almost perfect agreement was the highest for targetoid TP/HBP appearance at 76.5% and lowest for marked diffusion restriction at 45.5% (Fig. 3). The results of the subgroup of 21 studies using Cohen’s κ, which were similar to the values obtained for all included studies, are provided in Supplementary Table 4.
Substantial heterogeneity was observed across all LR-M imaging features (Cochran’s Q test, P<0.01; I2 ≥89.2%). The features showed no significant publication bias (P>0.05; Table 2 and Supplementary Fig. 1), except for peripheral washout (P=0.04) and necrosis or severe ischemia (P=0.03; Table 2 and Supplementary Fig. 1).
Of the 24 included studies, 22 reported inter-reader agreement for the assessment of MRI LR-M features. One reported combined CT and MRI results, and the other reported CT and MRI results separately. Therefore, the inter-reader agreement for the assessment of CT LR-M features was available in only one study:22 κ=0.70 (95% CI, 0.50-0.91) for rim APHE, 0.32 (95% CI, -0.10 to 0.81) for peripheral washout, 0.68 (95% CI, 0.52-0.85) for infiltrative appearance, and 0.66 (95% CI, 0.56-0.76) for necrosis or severe ischemia.
Subgroup meta-regression analysis
The results of the subgroup meta-regression analyses of targetoid features are summarized in Table 3. In the subgroup analysis of rim APHE, studies with a small observation size (<3 cm) showed significantly lower inter-reader agreement than those with a large observation size (≥3 cm; κ=0.59 vs. κ=0.78, P<0.01). Additionally, studies that used 1.5-T MRI showed poorer agreement than those that used 3.0-T MRI (κ=0.65 vs. κ=0.79, P=0.04). For peripheral washout, studies with a small observation size (<3 cm) exhibited poorer agreement than those with a large observation size (≥3 cm; κ=0.64 vs. κ=0.35, P<0.01). For the features of DCE and targetoid restriction, studies using 1.5-T MRI demonstrated significantly poorer agreement than those using 3.0-T MRI (DCE, κ=0.55 vs. κ=0.78, P=0.01; targetoid restriction, κ=0.49 vs. κ=0.70, P=0.02). For targetoid TP/HBP appearance, studies using 1.5-T MRI showed poorer agreement than those using 3.0-T MRI (κ=0.64 vs. κ=0.82, P=0.02), and studies with multiple readers (≥3) showed poorer agreement than those with two readers (κ=0.33 vs. κ=0.87, P<0.01).
The results of the subgroup meta-regression analysis for non-targetoid features are summarized in Table 4. For infiltrative appearance, marked diffusion restriction, and necrosis or severe ischemia, only MRI field strength was demonstrated as a contributing factor to significant differences in agreement (infiltrative appearance, 3.0-T, κ=0.83 vs. 1.5-T, κ=0.48, P=0.03; marked diffusion restriction, 3.0-T, κ=0.78 vs. 1.5-T, κ=0.39, P=0.04; necrosis or severe ischemia, 3.0-T, κ=0.77 vs. 1.5-T, κ=0.46, P=0.02).
DISCUSSION
For the 24 published studies with 5,163 hepatic observations, the overall pooled inter-reader agreement for the assessment of LR-M features was moderate to substantial. Most LR-M features showed substantial agreement, whereas peripheral washout (κ=0.52) and marked diffusion restriction (κ=0.49) showed moderate agreement. In addition, studies with small observation sizes, 1.5-T MRI scans, and multiple imaging readers were significantly associated with poor agreement for the assessment of LR-M features.
Our results regarding the pooled inter-reader agreement for the assessment of LR-M features compare favorably with the inter-reader agreement for assessment of the LI-RADS major features of non-rim APHE (κ=0.72), non-peripheral washout (κ=0.69), and enhancing capsule (κ=0.66).35 Although LI-RADS has shown excellent performance when used to stratify the probability of HCC based on the assigned categories,36 concerns about imperfect inter-reader agreement using LI-RADS have been raised.37 In particular, poor agreement for the LR-M criteria was reported in both research and clinical reading (ICC=0.46 for both).4 This may be important because LR-M lesions usually result in a biopsy for diagnostic confirmation and cannot be classified non-invasively.4 Given these imperfect agreements for the LR-M criteria, our study can be useful for understanding the causes of poor agreement because it provides the pooled inter-reader agreement for individual LR-M features and explores the factors associated with poor agreement in LR-M features. Furthermore, since our study included a large number of hepatic observations and involved a robust analysis, it should be helpful in understanding the current status and limitations of the CT/MRI LI-RADS v2018 LR-M criteria.
Of the LR-M features, poor agreement was found for peripheral washout and marked diffusion restriction (κ=0.52 and 0.49, respectively), with these features showing the lowest pooled κ among targetoid and non-targetoid features. LI-RADS has released a lexicon that provides a standardized vocabulary for liver imaging.38,39 In this LI-RADS lexicon, peripheral washout and marked diffusion restriction are defined as subtypes of washout that are mainly observed at the periphery and have an intensity higher than that of the liver on diffusion-weighted imaging that is not caused only by T2 shine-through, if marked in degree.38 However, these definitions have limitations because several details remain unclear. For example, it is not clear which area should be considered the “observation periphery,” or how “marked diffusion restriction” can be differentiated from “general restricted diffusion.” Furthermore, given the lower frequency of peripheral washout and marked diffusion restriction compared with other LR-M features (3% vs. 44% for rim APHE or 38% for targetoid TP/HBP appearance),10 it is possible that these two features are less familiar and experienced by radiologists. As all LR-M features are assessed qualitatively, some variability owing to the inherent subjectivity of imaging feature characterization may be unavoidable. Although the adoption of standardized lexicons has promoted clarity and consistency of communication in clinical practice and scientific literature, the situation is not perfect. Therefore, efforts to overcome inter-reader variability, including clearer definitions for qualitative assessment, illustrations of specific imaging features, and education, should continue.
The subgroup meta-regression analysis identified three significant factors associated with poor agreement in the assessment of LR-M features. Notably, for most LR-M features, 1.5-T MRI showed a poorer agreement than 3.0-T MRI. This result can be explained by the fact that 3.0-T MRI has a higher signal-to-noise ratio and lesion-to-liver contrast than 1.5-T MRI.40 Furthermore, meta-regression analysis revealed that a small observation size and large number of image readers were significant factors associated with poor agreement for LR-M features. In particular, for small HCC lesions, the definition of the peripheral portion may differ between readers, and ancillary features such as fat and blood products may prevent readers from detecting rim APHE.11 In addition, the interpretation may differ due to bias between readers, which might result from differences in training, experience, and frame of reference among readers. Therefore, as the number of image readers increases, variability in the reader experience is more likely to occur.
Our study has several limitations. First, substantial study heterogeneity was noted, which precludes the creation of solid meta-analytic summary estimates. To overcome this limitation, the causes of study heterogeneity (i.e., the causes of poor agreement) were robustly analyzed by performing a subgroup meta-regression analysis using various covariates. Second, during the second round of screening for eligibility, we excluded 19 articles because of insufficient data for extracting inter-reader agreement (Supplementary Table 5). As the meta-analysis for inter-reader agreement required variance as well as κ from each original study, the necessary information was sought by emailing the authors of studies that reported only the value of κ without variance, and if the variance could not be obtained, these studies were excluded. Third, most studies included in this meta-analysis evaluated LR-M imaging features using MRI, with only one study presenting results using CT. Consequently, drawing conclusions regarding the differences in inter-reader agreement between CT and MRI in LR-M evaluations based on the results of this study is challenging.
In conclusion, substantial inter-reader agreement was found for most LR-M features; however, the agreement for peripheral washout and marked diffusion restriction was limited. Additionally, a small observation size, the use of 1.5-T MRI, and multiple image readers contributed to the poor agreement for LR-M features. Therefore, the LI-RADS should continue to clarify and refine the definitions of individual LR-M features and provide educational programs, including a comprehensive manual with both schematic figures and clinical examples, to illustrate these features.
XML Download