Capsule endoscopy (CE) has been developed to obtain endoscopic imaging of the entire small bowel [1]. Since its introduction, clinical practice guidelines have been established for several conditions, including unexplained obscure gastrointestinal (GI) bleeding, small bowel Crohn’s disease, small bowel tumors, and other miscellaneous abnormalities [2,3]. Because endoscopy images are acquired by imaging sensors, most computer vision technologies [4] can be applied directly. Most of all, in the field of CE, various approaches based on artificial intelligence for computer-aided diagnosis have been undertaken to reduce the long review time. Here, we summarize deep learning–based works for CE and present possible direction of artificial intelligence for CE.

Since the early 2000s, the computer-aided decision support system (CADSS) has been researched extensively; as such, endoscopes can now take digital pictures [5]. CADSS has been designed to improve diagnostic accuracy by classifying abnormalities. In addition, a supportive system instead of direct decision-making support has emerged, including image quality enhancement, depth information extraction, and endoscopy localization. The ratio of the number of publications between conventional flexible endoscopy and CE regarding CADSS has been similar since 2007 [5].

Based on artificial intelligence, specifically computer vision and machine learning methodologies, various computational methods including algorithms for detecting hemorrhage and lesions, reducing review time, localizing capsules or lesions, and enhancing video quality have been proposed to improve efficiency and diagnostic accuracy [6]. For detecting hemorrhages and lesions, color and texture information is usually used as a distinctive feature [7-10]. Image features extracted from endoscopic images can be classified into target class using machine learning algorithms such as a support vector machine (SVM) [7-9], neural network [10], or binary classifier [11]. Although previous methods based on machine learning classifiers with invariant features showed concordant results for detecting various lesions, they have limitations related to insufficient training and testing databases and problems with specific feature design.

In early computer vision, image features such as corner [12] and edge [13] corresponding to scene structures were used to infer the class or 3D geometry of the target object. However, these primitive features are not invariant to various imaging condition changes (camera rotation and translation, illumination changes, occlusion, background clutter, and so on). To handle various changes, invariant features have been proposed that are robust to scale [14,15], affine [16], and local shape changes [17]. These handcrafted features have shown good performance for image-based recognition [18,19]. However, their results for very large-scale datasets are insufficient for practical application [20]. Since deep learning–based methods have shown much improved recognition performance [21-24], most computer vision and machine learning problems have been approached using deep learning.

Deep learning–based lesion detection and classification methods for flexible endoscopy have recently been presented [25]. The ability of computer-assisted image analysis with a deep learning–based method, more specifically convolutional neural networks (CNN), has been tested to detect polyps, a surrogate for adenoma detection rate. With 8,641 labeling datasets from the colonoscopies of over 2,000 patients, the method showed an accuracy of 96.4%. For polyp detection, the binary classification task (whether an input image contains at least one polyp) has been performed using CNN architectures such as VGG (Visual Geometry Group from Oxford) [22] and ResNet [24]. For polyp localization, a task involving localizing the polyps in the image, a variation of Darknet has been used [26].

Several methods based on deep learning have been proposed for CE in Table 1 [27-34]. Zou et al. proposed a CNN-based method to solve the classification problem of digestive organs in CE [27]. The problem has three possible classes: stomach, small intestine, and colon. Compared to conventional scale invariant feature transform (SIFT)– [14] and SVM-based approaches (90.31%), the proposed method showed an accuracy of 95.52% for 15K images from 25 patients. Similarly, Seguí et al. proposed a classification method of motility events such as turbid, bubbles, clear blob, wrinkle, and wall [28]. They obtained an accuracy of 96.01% for 100K and 20K training and testing dataset images, respectively. The previous approach of combining handcraft features such as gist [35], SIFT [14], and color only achieved 82.8% accuracy.

For detecting bleeding or hemorrhaging, deep learning–based approaches have demonstrated 99.9% accuracy for 2,850 positive images [29] and 100% accuracy for 390 positive images [30]. The sensitivity is over 99%. Because the color cue of hemorrhages is obvious, the accuracy and sensitivity are much higher than those for classifying problems of digestive organs [27] or motility events [28]. Among various deep learning networks, there is a non-negligible discrepancy between the highest performance network (GoogLeNet, 100%) and the lowest performance network (LeNet, 97.44%) [30]. To detect GI angiectasia, a CNN-based sematic segmentation algorithm was proposed [31]. From 200 capsule endoscopies, 20,000 normal frames and 2,946 frames with vascular lesions were extracted. To avoid overfitting, they used 600 images for training and another 600 images for testing by excluding successive frames of the same lesions. This work obtained a sensitivity of 100% and a specificity of 96%.

For detecting polyps, Yuan and Meng proposed a stacked sparse autoencoder–based approach [32]. Compared to other machine learning–based previous works, they achieved an accuracy of 98% for 4,000 images from 35 patients. Their method also classified normal images into turbid, bubble, and clear types. For detecting hookworms, He et al. proposed a novel edge extraction network to capture their characteristics [33]. From 440K images of 11 patients, the accuracy and sensitivity of the proposed network were 88.5% and 84.6%, respectively. When previous networks such as Alexnet [21] and GoogLeNet [23] were applied, the accuracy was higher (96.0% and 93.7%, respectively); however, the sensitivity was much lower (48.1% and 77.1%, respectively). Therefore, a novel network design for the specific problem is very important to obtain relevant results.

Iakovidis et al. presented a three-phase approach: a weakly supervised CNN for abnormality classification, deep saliency detection to detect salient points, and iterative cluster unification to localize GI anomalies [34]. They tested their proposed method for two datasets: a larger dataset (D1) with 10K images from more than 1,000 volunteers and a smaller dataset (D2) with 2,352 images. Similar to Leenhardt et al. [31], they used 465 and 233 images for D1 training and testing, respectively, and 852 and 344 images for D2 training and testing, respectively. Compared with other deep learning– and machine learning–based previous works, their method showed improved results for D1 because they designed a complex multi-stage architecture for CE. However, their results for D2 were comparably worse than other methods, especially the low sensitivity (36.2%). We believe that the D2 is too small for the proposed deep learning network to be learned effectively.

Although recent works based on deep learning have shown better performance than previous works based on handcrafted features [27,28,33,34], there are contrary cases in which the method on handcrafted features was better for other problems [36]. For medical modality classification, Harris corner [12] with SIFT [14] and local binary pattern [37] descriptors showed better classification results than the CNN-based method because the number of images in the dataset for various modalities is insufficient to train a deep architecture [36]. The need for a sufficiently large database is one of the limitations of deep learning–based approaches. Overfitting is another crucial issue for deep learning–based approaches. This issue becomes more severe in cases of small databases. Although there are several approaches to mitigate overfitting in deep learning [38,39], it remains an important problem for practical use. Overfitting should be considered for medical applications, which can be reduced by cost function regularity, data augmentation, relevant data selection, and other factors.

Here we reviewed recent deep learning–based approaches for CE that have been applied to various problems such as scene classification and the detection of bleeding/hemorrhage/angiectasia, polyp/ulcer/cancer, and hookworms. Using large datasets from dozens of patients, they achieved much higher accuracy and sensitivity rates, sometimes close to 100%, compared to precious machine learning–based methods.

Because collecting databases for CE is difficult, more effective and generalized methods with the cooperation of many physicians and artificial intelligence engineers are required. Similar to other areas such as computer vision and robotics, the deep learning–based methodology will become more convincing and widely used. For medical applications, however, there are other bottlenecks such as dataset gathering, determination in terms of clinical aspects, and practical usage of computer-aided methods. Several research topics for CE remain, such as capsule localization, image enhancement, and reducing review time. One of the main drawbacks for CE is the lack of prospective trials to verify the accuracy of the computer-aided diagnosis. Since retrospective studies have reported more meaningful outcomes such as real-time image-based analysis during colonoscopy [40,41], prospective research using CE is also very important for clinical applications.