A number of radiomics features can be extracted from a given volume of interest (VOI) drawn on two-dimensional (2D) images or three-dimensional (3D) volume data. Radiomics features can be divided into morphologic features, histogram features, textural features, and higher-order features.

The radiomics analysis of medical images involves multiple processes, including image preprocessing, segmentation, feature extraction, feature selection, and classification.

Image preprocessing is an important step for achieving valid and reproducible radiomics features. Image normalization may be required to standardize the gray-scale pixel value, and it can be performed based on the histogram distribution of pixel values or internal reference values (i.e., spleen signal value). Since textural and higher-order features are dependent on pixel dimensions, images with non-isometric pixels or variable resolutions may lead to invalid results. In these cases, the image resolution should be standardized by the resampling the images at a fixed isometric resolution. After image preprocessing, the segmentation of VOIs is performed manually or by using an automatic segmentation algorithm to select the volume or area for which the radiomics features are extracted.

Radiomics features can be extracted in 2D or 3D using in-house software (6910111213) or commercial software (1415). The number of extracted features can be variable, largely depending on the number of textural features and the number of filters used for extracting higher-order features.

A larger number of extracted features do not necessarily indicate better-quality measurements. Since radiomics features are highly correlated with each other (16), the analysis of high-dimensional features may lead to problems of multicollinearity and overfitting. A recent phantom study revealed that the information provided by multiple radiomics features could be summarized using only 10 features because of redundancy (16).

Feature selection is a process performed to reduce the dimensionality of features by selecting informative and reliable features and excluding redundant features among the extracted features. Classification is a process used to build a classifier or prediction model using the selected feature to perform a given classification or prediction task. Feature selection and classification can be performed together as a single process or separately using different algorithms. Unreliable features may be excluded prior to feature selection and classification, based on the results of inter- or intra-observer agreement or test-retest repeatability analyses (6111718192021). To reduce redundancy in the features, informative features showing a high dynamic range may be selected among the correlated features in hierarchical feature clustering (1822). Traditional statistical methods may not be successful in dealing with high-dimensional radiomics features (i.e., too many variables relative to the number of observations). A number of machine learning methods have therefore been used for feature selection and/or classification (10212324). Among the methods for feature selection and classification, regression with Ridge, least absolute shrinkage and selection operator (LASSO), and elastic net regularization have been commonly used (61011121317192526), likely because these algorithms allow for the development of a regression model that is more familiar to radiologists than other machine learning classifiers. These regression analyses incorporate regularization and penalization algorithms for correlated variables; the LASSO regression method is robust for feature selection, but the Ridge regression is more effective in dealing with multicollinearity. Elastic net regression takes advantage of both methods (627). Other commonly used classification methods include regression, support vector machine (SVM), decision tree, and random forest. In machine learning, hyperparameters to control the machine learning process need to be optimized to different data patterns. Following hyperparameter optimization, the machine learning algorithm is trained through the learning process using given training data (28). Figure 5 schematically depicts the development process of a radiomics classification model.

Chronic liver disease is accompanied by changes in liver volume, morphology, and texture. Several recent studies have shown the potential value of radiomics as a method for comprehensive and objective analysis of such changes in the liver using imaging examinations. Park et al. (6) developed a radiomics fibrosis index based on radiomics features extracted from gadoxetic acid-enhanced hepatobiliary phase magnetic resonance (MR) images. They demonstrated that the radiomics fibrosis index had a high diagnostic performance in staging liver fibrosis (area under the receiver-operating curve [AUROC], 0.89–0.91) and significantly outperformed the normalized liver enhancement and serum fibrosis indices. Liu et al. (12) reported the feasibility of CT-based radiomics analysis for the diagnosis of clinically significant portal hypertension. These authors devised a model based on texture features, morphologic features, and the liver and spleen volumes with the hepatic venous pressure gradient as the reference standard. The performance of this model was significantly better than were those of models using liver stiffness measurements as well as other radiologic and clinical indices (12). Several exploratory studies have indicated the potential of radiomics of multiparametric ultrasound (21) and histogram features of CT images (2930) in staging liver fibrosis and in diagnosing nonalcoholic steatohepatitis. However, the results of these studies were not conclusive because of the small study populations and the lack of proper validation (212930).

Radiomics has been applied to determine the prognosis of hepatocellular carcinoma (HCC) after radiofrequency ablation (3132), surgical resection (1322253133), and liver transplantation (20). Zheng et al. (13) developed nomograms incorporating CT-based radiomics and clinical variables to predict recurrence-free and overall survival outcomes after surgical resection of solitary HCC and reported that these nomograms had better prognostic performance than traditional staging. Kim et al. (22) devised radiomics models for predicting the early and late post-surgical recurrence of HCC using gadoxetic acid-enhanced MRI, incorporating variable extents of peritumor border extension. In that study, a radiomics model with 3-mm or 5-mm peritumoral border extension showed a higher prediction performance than the models without a border extension, indicating that the features of a peri-tumor liver parenchyma are important for predicting early or late recurrence in HCC patients. Since microvascular invasion (MVI) is one of the most important prognostic factors for HCC after surgery (343536), several studies have evaluated the potential of using radiomics to predict it (10242637). Xu et al. (24) developed a prediction model combining a CT-based radiomics score, radiologist image analysis, and laboratory findings and demonstrated a high accuracy (AUROC, 0.889) for predicting MVI in a test dataset. In their study, however, subsequent decision curve analysis failed to demonstrate the incremental value of the radiomics score in comparison with conventional visual image analysis. Two prior studies have reported the incremental value of a CT-based radiomics model in predicting lymph node metastases in patients with cholangiocarcinoma, and noted that incorporating a radiomics signature into the CT-reported lymph node status improved the detection of lymph node metastasis (1117). A recent study (9) has also demonstrated the potential role of radiomics features extracted from gadoxetic acid-enhanced hepatobiliary MR images in assisting with precision immunotherapy of HCC. This study showed that a model combining radiomics and clinical variables accurately predicted the immune-score, which is known to be associated with the therapeutic response to an immune checkpoint blockage (9).

Convolutional neural network (CNN) is the most popular type of deep learning architecture in medical imaging analysis (4142). A CNN consists of an input layer, hidden layers, and the output layer. The hidden layers may include convolution and pooling layers and fully connected layers. Convolution and pooling layers extract high-dimensional manageable features from given images, which is conceptually similar to the feature extraction process used in radiomics analysis. Convolution operations generate feature maps using a group of filters, followed by activation functions typically using a rectified linear unit. Activation functions add nonlinearity to the outputs of convolutions, allowing the selection of features to pass through to the next layer. Pooling operations reduce the resolution of the feature maps to gain computational performance, obtain spatially invariable features, and reduce the chance of overfitting (4143). The fully connected layers integrate and transform all of the features fed from the convolution and pooling layers into a vector form. The output layer then returns a categorical distribution for class probability through a softmax function. The details of deep learning and CNN can be understood further from previous review articles (4142). Figure 6 schematically presents the architecture and training process of a CNN algorithm.

The training of a deep learning algorithm is usually performed with supervised learning using labeled training data. A deep learning algorithm typically requires large volumes of high-quality ground truth training data, although the amount of required data may vary for different deep learning algorithm tasks: an algorithm for a segmentation task may require a smaller dataset, while a classification task requires a much larger dataset (44). When a training dataset is not sufficiently large, data augmentation may be used to enlarge it artificially, which is performed through random transformation of original images by adding random noise, flipping, or rotation (41). Data augmentation may also be required to overcome the potential problems of data imbalance. If the size of the training data is imbalanced across different classes, a classification algorithm may have poor classification accuracy for the minority classes (45). This may be prevented by data augmentation for those classes. Datasets for the development and validation of a deep learning algorithm typically consist of training, validation, and test datasets. The data available for the development of the algorithm may be divided into training and validation datasets. The validation dataset is used for monitoring the performance of the algorithm during the training process and/or comparing multiple models based on different CNN architectures or hyperparameters. Once the final model is selected and all its parameters are fixed, its performance is evaluated in the test dataset. The test dataset is used only at the final step of the study to report the final model performance (41). A deep learning algorithm is trained by adjusting network weights. Starting from a random initial configuration, parameters are adjusted to find a set of parameters that perform best on the training dataset. During the training phase, the output of the algorithm is compared with the ground truth by using a loss function that quantitatively measures the error in the prediction in comparison with the ground truth. The error is then back propagated to optimize network weights (Fig. 6). The training phase continues until the loss function reaches a minimum.

Liver segmentation has direct clinical applications, including liver volume measurement, which is important in pre-operative planning for liver resection (4647), determination of the radiation dose in liver tumor radioembolization, and measurement of quantitative indices such as the proton density fat fraction (PDFF) from the whole liver (48). Notably, however, liver segmentation is labor-intensive and time-consuming, which limits its usage in clinical practice. Thus, deep learning has been applied for automated segmentation of the liver. The U-net architecture is most commonly used for segmentation tasks (49) and consists of a series of contracting and expanding layers that extract and process features from input images and return a pixel-wise probability map. The segmentation performance is typically evaluated using the Dice similarity score (DSS), defined as 2 × true positive pixels / [2 × true positive pixels + false negative pixels + false positive pixels]. Some prior studies have reported the use of deep learning algorithms for automated liver segmentation on CT or MRI (5051525354), and some have utilized a deep learning algorithm combined with image processing methods (5052). All of these studies reported high performance values in liver segmentation, with the reported DSS values ranging from 0.92 to 0.95 (5051525354). Recently, Wang et al. (48) demonstrated the feasibility of generalized CNN, which can be used for liver segmentation on CT scans and various MRI sequences using the transfer learning technique. They reported DSS values ranging from 0.92 to 0.95 for liver segmentation on CT and MR images. Furthermore, these authors demonstrated a close agreement between the PDFF values measured using deep learning-based automatic liver segmentation and those measured by manual liver segmentation, indicating the potential role of deep learning-based liver segmentation for automatic measurement of quantitative indices from the whole liver. Despite these promising results, however, further clinical validation may be required for the actual clinical application of deep learning algorithms for automated liver segmentation. For example, algorithm performance should be evaluated in a healthy liver, fatty liver, and in chronic liver disease and liver cirrhosis. With continued improvements in deep learning-based organ segmentation methods, it is expected that fully automated liver segmentation would become clinically available in the near future.

A few deep learning algorithms for liver fibrosis staging have been reported, to date. Liu et al. (55) proposed sequential algorithms to diagnose cirrhosis using ultrasound images, which first detect liver capsules on the images by using a sliding window detector, extract features from image patches by using a CNN algorithm, and finally classify an image as indicative of cirrhosis or not by using an SVM. In that report, CNN was used only for feature extraction whereas classification was performed with the SVM because of the small amount of training data. Yasaka et al. (56) developed CNN algorithms for liver fibrosis staging using cropped CT images and cropped gadoxetic acid-enhanced hepatobiliary phase MR images (57). They reported area under the curves (AUCs) of 0.73–0.76 for the CT-based algorithm and 0.84–0.85 for the MRI-based algorithm in staging liver fibrosis. However, the use of a small test dataset (100 patients) and lack of any external validation limited the generalizability of their study results. Choi et al. (45) reported the use of a deep learning algorithm for fully automated liver fibrosis staging using portal venous phase CT images. Using a large training dataset (7491 patients) and internal and external test data (891 patients), these authors reported a high accuracy (AUCs, 0.95–0.97) of the deep learning algorithm in liver fibrosis staging, surpassing that of the serum fibrosis indices and visual image analyses by radiologists. A recent multicenter prospective study reported a higher accuracy (AUCs, 0.97–0.98) with a deep learning algorithm using cropped 2D shear wave elastographic images in staging liver fibrosis in comparison with liver stiffness measurement results (8).

The feasibility of using deep learning for the diagnosis and grading of fatty liver disease using ultrasound images has been evaluated in several previous reports (585960). Although these prior studies demonstrated the technical feasibility of deep learning, its clinical applicability has not been well proven because of the small size of the test data, lack of external validation, and the use of a less reliable reference standard (i.e., ultrasound determined fatty liver grade).

Vorontsov et al. (61) have reported the use of a deep learning algorithm for the automatic detection and segmentation of malignant liver tumors on CT images. In a small test dataset (26 CT examinations) in that study, the algorithm showed high accuracy in detecting liver lesions larger than 2 cm with a sensitivity of 85% and positive predictive value of 94%, whereas it was not accurate in the detection of small lesions (sensitivity, 10% for lesions < 1 cm) or in automatic tumor segmentation (DSS of 0.14–0.68). Schmauch et al. (62) also described the technical feasibility of applying deep learning to the detection of focal liver lesions using ultrasound images. The potential utility of deep learning for the classification of focal hepatic lesions has now been evaluated in several studies, all of which devised deep learning algorithms to classify liver lesions into five to six predefined categories based on manually cropped CT or MR images containing these lesions (6364). Yasaka et al. (63) developed an algorithm for classifying liver masses using multi-phasic CT images and reported an accuracy of 84% in the test dataset. Hamm et al. (64) reported the results of algorithms based on multiphasic MRI, describing an accuracy of 90% for lesion diagnosis and 92% for lesion categorization based on the liver imaging reporting and data system. The same researchers (54) also demonstrated the feasibility of deep learning in identifying individual radiologic features of focal hepatic lesions on MR images, reporting a sensitivity of 82.9% and positive predictive value of 76.5% for the algorithm. Despite these promising results, however, all prior studies on the application of deep learning to liver lesion detection and characterization are considered preliminary. These earlier reports focused mainly on the technical feasibility of deep learning, since the algorithms used involved data processes not suitable for a real clinical workflow (e.g., image cropping by radiologists) and were not fully validated using a large-scale external dataset.

Deep learning has now been used for automatic evaluation of image quality (6566). Ma et al. (65) reported a deep learning algorithm to identify technically optimal portal venous phase CT images. Esses et al. (66) described an algorithm to discriminate diagnostic and nondiagnostic T2-weighted MR images. With further improvements, these techniques may be clinically usable for real-time scanning optimization through automatic image quality monitoring. Recent research findings have further suggested the potential utility of deep learning as a method to improve MR image quality (6768). Tamada et al. (68) presented a method to reduce respiratory motion artifacts in gadoxetic acid-enhanced arterial phase MR images using a CNN algorithm. Liu et al. (67) developed a deep-learning-based MR image reconstruction algorithm by adopting generative adversarial networks (GANs). These authors demonstrated that their GAN-based reconstruction algorithm produced superior image quality in comparison with a reconstruction algorithm based on compressed sensing and parallel imaging. This suggested the potential of deep learning-based image reconstruction combined with data under-sampling for fast MRI.