Endoscopy has played an important role in gastrointestinal (GI) tract examination because it enables clinicians to directly observe the GI tract. However, the accuracy of its diagnostic results is limited by the experience of the practitioner and complex environmental factors of the GI tract [1]. Therefore, there is an increasing interest in the field of endoscopic imaging regarding a method for improving the accuracy of diagnosis.

Artificial intelligence (AI) based on deep learning is making remarkable progress in various medical fields. Endoscopic imaging is one of the most effective applications of AI-based analytics in the medical field [2,3]. A convolutional neural network (CNN) comprises a deep learning architecture that is best known for its application in imaging data analysis [4]. Recently, several authors have reported successful application of CNN to GI-endoscopic image analysis [5-8]. Among various diseases of the GI tract, early gastric cancer (EGC) has certain special characteristics. This review focuses on the application of lesion- based CNN in EGC.

CNN technology is based on the principle that the visual cortex of the human brain is responsible for the recognition of images. A CNN can be divided into components that extract the features of images and those that classify data. The feature extraction part comprises iterations of the convolutional and pooling layers. The convolutional layer is typically composed of a filter and an activation function. The filter performs a convolution operation to extract features from input data. When the input image is processed into consecutive convolutional layers, the filters are accumulated. This process creates increasingly descriptive and sophisticated feature detectors. When the feature map is extracted, the activation function converts it to quantitative non-linear values. Overfitting occurs owing to the increased number of features; therefore, a new layer is obtained by resizing via the pooling layer. The key features are extracted from the image through iterations of the convolutional and pooling layers. Subsequently, classification is performed through the fully connected layers, which are the last layers of the CNN [9]. Fig. 1 shows an example of the classification of GI-endoscopic images by a CNN.

CNN was first developed by Fukushima in 1980 [10]. In 1995, Lo et al. reported the detection of lung nodules using a CNN [11]. However, at the time, it was forgotten because of limitations of computational power. About 17 years later, a CNN overcomes these limitations by improving computation performance with the development of graphical processing units and applying a deep learning method. Deep learning allows more efficient learning through multiple intermediate layers between the input and output layers. In 2012, Krizhevsky et al. designed a CNN with deep learning method (AlexNet, designed by the SuperVision group) and won the ImageNet Large Scale Visual Recognition Challenge with an overwhelming performance [9]. Since then, numerous authors have studied medical image analysis using CNN with deep learning method [12-14].

Most examinations and diagnoses of GI tract diseases are performed using endoscopy [1,15]. Therefore, accurate analysis of endoscopic images is important, and the application of CNN can be considerably useful. CNN has been applied for the detection and pathological classification of colon polyps [16-19]. Karnes et al. added narrow-band imaging (NBI) to improve the accuracy of polyp detection [20]. Byrne et al. developed a system for real-time assessment of colon polyps in a video format [19]. Additionally, CNN was applied in upper endoscopy to detect gastritis caused by Helicobacter pylori infection [21] and classify the subtypes of Barrett’s esophagus [22]. Moreover, it is currently being applied for the detection and classification of GI tract cancer, including cancer of the esophagus, stomach, and colon [5,6,23].

The role of endoscopic examination in the diagnosis of EGC includes the characterization and detection of lesions. Awareness of the signs of suspicious lesions is important. However, some EGC lesions can be difficult to detect because of extremely subtle changes in the endoscopic examination [24]. Furthermore, EGC detection becomes complex owing to chronic inflammation or intestinal metaplasia of the surrounding mucosa. Recently, some authors reported that image enhancement technology, such as magnifying NBI, improves the ability to identify mucosal abnormalities [25-27]. However, the enhancement technology is not widely used currently because of the lack of additional experience, special equipment, and time.

Accurate detection of EGC requires both knowledge and technical expertise [28]. It is required that endoscopic techniques do not create blind spots during examination. Additionally, the air inflation should be adjusted to adequately reveal the lesions. These technical aspects are reflected in the endoscopic image. However, owing to the nature of endoscopic examination, it is impossible to standardize all these conditions because of variations in both the practitioner and the patient’s condition. The lack of standardization in imaging can be an important limitation for research using images. Furthermore, it is impossible to fully standardize the image acquired via endoscopy. Therefore, it is important to select high-quality images that adequately reveal the features of the lesion.

Pre-treatment prediction of the invasion depth (T-staging) has becoming increasingly important because it is an essential factor in determining the treatment method for EGC [29]. Some authors have reported that conventional endoscopy is comparable to endoscopic ultrasound for predicting the invasion depth of EGC [30]. Abe et al. reported a depth-predicting score for the differentiated-type EGC [31]. Tumors with sizes greater than 30 mm, remarkable redness, uneven surface, and margin elevation were associated with deep submucosal cancers [31]. However, this approach can be challenging because these features are extremely subtle in EGC. Therefore, there has been an increasing interest in the field of medical imaging regarding the modalities for predicting the depth of EGC.

Gastric cancer shows greater histological heterogeneity than other cancers. Even tumors confined to the mucosa show histologic heterogeneity, which tends to increase with deeper invasion and increased tumor diameter [32]. Such mixed histology has been reported to be associated with more aggressive behavior than other histological types [33,34]. Till date, the histological heterogeneity of EGC mucosa cannot be accurately predicted without biopsy. While evaluating EGC, it is important to identify the entire lesion owing to the heterogeneity of EGC.

Fundamentally, deep neural networks are black boxes, indicating that it is not known how the network produces the output. Therefore, the results are analyzed using statistical methods. For example, various measures, such as sensitivity, specificity, and accuracy, are used to analyze the results and identify problems in the network. However, we can only estimate how the network performs judgements, while the forwarding process of the network remains unclear. This limitation can lead to critical errors in the learning process involving endoscopic images. For example, in the case of AI that is focused only on classification for EGC diagnosis or depth prediction, basic normal structures, such as the pyloric channel or esophagogastric junction, might be incorrectly recognized as lesions. Therefore, the validity of the process needs to be confirmed instead of merely accepting the classification result of the AI. To design a more reliable deep learning system, there has been an increasing interest in the interpretability of AI. Selvaraju et al. proposed the gradient-weighted class activation mapping (Grad-CAM) approach for producing “visual explanations” for the decisions reached by a CNN [35]. Grad-CAM produces a visual explanation via gradient-based localization of deep learning networks. All neural networks learn by back-propagating a gradient, using the common gradient as a weight applied to the feature of the layer that needs to be visualized. Application of the Grad-CAM method confirms that the network is properly trained and the entire lesion is evaluated within the endoscopic image. During the training process, the CNN learns about the localization errors and constructs heatmaps for those regions where it focuses. The blue and red colors on the Grad-CAM indicate lower and higher activation values, respectively (Fig. 2). Fig. 3 shows an example of the lesion-based CNN algorithm with the Grad-CAM method. The Grad-CAM of the last convolutional layer was used to measure the Grad-CAM loss. Activated regions with high values were more often used as visual features in the following fully connected layers. By applying the Grad-CAM method to EGC, the entire lesion can be evaluated to reduce localization errors and improve performance. Yoon et al. named this process, “lesion-based CNN” [36]. They reported that it showed better performance than other CNNs in the detection of EGC and depth prediction. The Grad-CAM loss was applied to the existing cross-entropy loss to reduce localization errors. During training, the model was taught to properly detect EGCs while activating the EGC regions by simultaneously optimizing the cross-entropy and Grad-CAM losses. However, even with the same type of CNN, the result may vary depending on whether the Grad-CAM method is applied [36]. There are some limitations in this system. Inaccuracy of the hand-drawn margin of EGC is possible—this is not a pixel-by-pixel operation; therefore, there is a potential for error. Additionally, in some complex cases, even experts find it difficult to accurately determine the margin of EGC through conventional endoscopy. In the future, a large amount of learning data and the application of image enhancement techniques, such as magnifying NBI, should be considered to improve the lesion-based CNN approach.

CNN is an effective method to improve endoscopic image evaluation. The application of CNN will significantly enhance the effectiveness of endoscopy. To maximize clinical usefulness, it is important to determine the optimal method of CNN application. Lesion-based CNN using Grad-CAM is one of the most effective AI-based methods for EGC diagnosis. However, the application of AI to endoscopic images should not remain restricted to a level where its image classification performance is superior to that of humans—achieving optical biopsy should be the final objective of this approach. This can be achieved by improving the performance, based on the characteristics of each organ and disease.