PDF
Citation
Print
INTRODUCTION
Accurate positioning of cephalometric landmarks is one of the most important steps in successful cephalometric analyses. Since the location and visibility of some anatomic landmarks are highly influenced by superimposition of the anatomical structures in the face between the right and left sides,1,2 it is not easy to identify these anatomic landmarks consistently and accurately.
For the last several decades, clinicians have manually indicated the cephalometric landmarks and measured several angles and distances between these landmarks to assess dentofacial deformities.3 Although this manual cephalometric analysis has been substituted with digital cephalometric analysis,4,5 the process is still laborious, time-consuming, and sometimes inaccurate in detection of cephalometric landmarks.3,6-8
Recently, research on automatic detection of cephalometric landmarks using artificial intelligence (AI) with convolutional neural networks (CNNs) has gained popularity.1-3,9-11 These studies have focused mainly on automatic detection of cephalometric landmarks and reported that most cephalometric landmarks were detected within a 2-mm range of accuracy.1,10 However, these approaches still require further measurements of cephalometric parameters including distance, angle, and ratio. Although Kunz et al.11 developed an AI algorithm to analyze 12 cephalometric parameters, they did not make a one-step automated orthodontic diagnosis tool in practice. Therefore, it is necessary to develop a one-step automated orthodontic diagnosis algorithm based on a CNN to avoid the need of additional measurements of cephalometric parameters.
In terms of a one-step CNN algorithm for classification of skeletal discrepancies, Yu et al.8 reported > 90% accuracy, sensitivity, and specificity for diagnosis of the sagittal and vertical skeletal discrepancies in three models (Models I, II, and III). However, they intentionally excluded some portion of the data adjacent to the classification cutoff with intervals of 0.2 standard deviations (SDs) in Model II and 0.3 SDs in Model III in the test set.8 As a result, Models II and III showed a significant increase in the values for accuracy, sensitivity, and specificity compared to Model I.8
The major limitations in previous studies can be summarized as follows:1-3,8-11 (1) Most studies used lateral cephalograms from only one or two hospitals, not from nationwide several different hospitals which had different machine types, radiation exposure conditions, sensors, and image conditions; (2) No study has simultaneously reported dental and skeletal discrepancies using a one-step automated classification algorithm; and (3) If some portion of the data adjacent to the classification cutoff were excluded in the test set, there would be issues in the continuity of the test set and an exaggerated increase in accuracy. Therefore, the purpose of this study was to investigate the accuracy of a novel one-step automated orthodontic diagnosis model for determining anteroposterior skeletal discrepancies (APSDs: Class I, Class II, and Class III), vertical skeletal discrepancies (VSDs: normodivergent, hyperdivergent, and hypodivergent), and vertical dental discrepancies (VDDs: normal overbite, open bite, and deep bite) using a CNN and lateral cephalogram images with different qualities from nationwide 10 unrelated dental hospitals in Korea.
MATERIALS AND METHODS
Description of the dataset
A total of 2,174 lateral cephalogram images were retrospectively obtained from the Departments of Orthodontics in nationwide 10 hospitals including Seoul National University Hospital (SNUDH), Kooalldam Dental Hospital (KADH), Ajou University Dental Hospital (AJUDH), Asan Medical Center (AMC), Chonnam National University Dental Hospital (CNUDH), Chosun University Dental Hospital (CSUDH), Ewha University Medical Center (EUMC), Kyung Hee University Dental Hospital (KHUDH), Kyungpook National University Dental Hospital (KNUDH), and Wonkwang University Dental Hospital (WKUDH) in Korea. The inclusion criteria were Korean adult patients who underwent orthodontic treatment with/without orthognathic surgery between 2013 and 2020. The exclusion criteria were (1) patients who were in childhood and adolescence and (2) patients who had mixed dentition. All datasets were strictly anonymized before use. The study protocol was reviewed and approved by the Institutional Review Board of SNUDH (ERI20022), Korean National Institute for Bioethics Policy for KADH (P01-202010-21-020), Ajou University Hospital Human Research Protection Center (AJIRB-MED-MDB-19-039), AMC (2019-0927), CNUDH (CNUDH-2019-004), CSUDH (CUDHIRB 1901 005), EUMC (EUMC 2019-04-017-003), KHUDH (D19-007-003), KNUDH (KNUDH-2019-03-02-00), and WKUDH (WKDIRB201903-01).
Lateral cephalogram images, 1,993 from two hospitals, were used for the training set (n = 1,522) and internal test set (n = 471), and 181 from eight other hospitals were used as the external test set to validate our model (Figure 1). Table 1 summarizes information on the product, radiation exposure condition, sensor, and image conditions in each hospital, which showed diverse conditions.
Setting a gold standard for the diagnosis of APSDs, VSDs, and VDDs
After detection of the cephalometric landmarks including A point, nasion, B point, orbitale, porion, gonion, menton, sella, maxilla 1 crown, maxilla 6 distal, mandible 1 crown, and mandible 6 distal by a single operator (SY), the cephalometric parameters including A point-Nasion-B point (ANB) angle, Frankfort mandibular plane angle (FMA), Jarabak’s posterior/anterior facial height ratio (FHR), and overbite were calculated using V-Ceph 8.0 (Osstem, Seoul, Korea) to set a gold standard.
All cephalometric images were classified into the three classification groups by a single operator (SY) as follows. For classification of APSDs, we defined the ANB value between –1 SD and 1 SD from the ethnic norm of each sex12 as skeletal Class I; > 1 SD as skeletal Class II; and < –1 SD as skeletal Class III. For classification of VSDs, we combined FMA and FHR values from the ethnic norm of each sex12 for training. First, we normalized the FMA and FHR values by using the SD values. Second, the FHR values were flipped due to an opposite sign compared to the FMA values. Third, the values of FMA and flipped FHR were added because each are regarded as having equal weights. Fourth, the mean and SD values were obtained for classification into three groups. Then, we defined the values between –1 SD and 1 SD from the mean as normodivergent pattern, > 1 SD as hyperdivergent pattern, and < –1 SD as hypodivergent pattern. For classification of the VDDs, we defined the overbite value between 0 mm and 3 mm as a normal overbite, > 3 mm as a deep bite, and < 0 mm as an open bite (Tables 2 and 3).
To assess intra-examiner reliability, all classifications of APSDs, VSDs, and VDDs were performed again after one month by the same investigator (SY). Since the minimum sample size13 was suggested as 49 from a 3 × 3 Cohen’s kappa agreement test, 100 images were randomly selected to classify APSDs, VSDs, and VDDs. Cohen’s kappa agreement test showed an “almost perfect” agreement (kappa value; 0.939 for APSDs, 0.984 for VSDs, and 0.907 for VDDs).14 Therefore, the first classification results were used for further statistical analysis.
To evaluate inter-examiner reliability, the same images used to assess intra-examiner reliability were selected. Classifications of APSDs, VSDs, and VDDs were performed by the other investigator (KL). Cohen’s kappa agreement test showed an “almost perfect” agreement for APSDs and VSDs (kappa value; 0.985 for APSDs, 0.919 for VSDs) and “substantial’ agreement” for VDDs (0.601).14
Preprocessing of the data
Augmentation techniques including cropping, padding, spatial transformations, and appearance transformation were conducted in real time.
Model architecture (Figure 2)
As the backbone of the model, DenseNet-169 pre-trained with weights of the ImageNet dataset was used with group normalization (GN).15-20 After the global average pooling (GAP) of the backbone, ArcFace was added in parallel with the softmax layer in order to overcome imbalanced data sets and obtain discriminative features during training.21
After training, the ArcFace head was removed, and inference was implemented using only the softmax layer as a basic CNN classifier. Because sex was included as a classification criterion of APSDs and VSDs, the one-hot vector about sex was concatenated with the feature vector after GAP.
Model training (Figures 1 and 2)
Training for APSDs, VSDs, and VDDs was performed using only a gold standard determined by a single operator (SY), not by measurement of cephalometric parameters including ANB, FMA, FHR, and overbite.
Model testing
After training was completed, one-step classification was performed with both the internal and external test sets to validate the performance of the constructed model. It took 55 seconds (sec) to diagnose the internal test set (0.1168 sec per lateral cephalogram) and 22 sec to diagnose the external test set (0.1215 sec per lateral cephalogram). The results for the internal and external test sets were compared with gold standard diagnostic data.
Analysis method
t-stochastic neighbor embedding (t-SNE)
Since this technique can visualize high-dimensional data by giving each datapoint a location in a two or three-dimensional map, it was used to check the feature distribution of the training set, internal test set, and external test set after GAP layering.24 In each diagnosis, the labels of ground truth (GT) and prediction (PD) were set to check the distribution of each data set.
Gradient-weighted class activation mapping (Grad-CAM)25
As this technique can produce visual explanations of AI models, it can show the regions where the AI focuses for PD. It was used to confirm the regions where our model mainly focused on the diagnosis of APSDs, VSDs, and VDDs.
RESULTS
Metrology distribution of the APSDs, VSDs, and VDDs per dataset (Figure 3)
The continuity of the dataset between the normal groups (Class I in APSDs, normodivergent pattern in VSDs, and normal overbite in VDDs) and the other two groups (Class II and III in APSDs, hyperdivergent and hypodivergent patterns in VSDs, and open bite and deep bite in VDDs) was confirmed.
Accuracy and AUC of the internal test set in binary ROC analysis (Table 4 and Figure 4)
In APSDs, Class III had the highest accuracy and AUC (0.9372 and 0.9807, respectively), followed by Class II (0.8972 and 0.9533, respectively) and Class I (0.8488 and 0.9212, respectively). In VSDs, hypodivergent pattern had the highest accuracy and AUC (0.9346 and 0.9824, respectively), followed by hyperdivergent pattern (0.9019 and 0.9730, respectively) and normodivergent pattern (0.8365 and 0.9186, respectively). In VDDs, open bite had the highest accuracy and AUC (0.8730 and 0.9475, respectively), followed by deep bite (0.8637 and 0.9286, respectively) and normal overbite (0.7376 and 0.8177, respectively).
In APSDs and VSDs, the total accuracy reached nearly 0.9 and the total AUC exceeded 0.95 (0.9517 and 0.9580, respectively). However, VDDs showed a relatively lower total accuracy (0.8248 vs. 0.8944 and 0.8910) and total AUC (0.8979 vs. 0.9517 and 0.9580) than APSDs and VSDs.
Accuracy and AUC of the external test set in binary ROC analysis (Table 4 and Figure 4)
In APSDs, Class III had the highest accuracy and AUC (0.9525 and 0.9930, respectively), followed by Class II (0.8796 and 0.9601, respectively) and Class I (0.8320 and 0.9042, respectively). In VDDs, open bite had the highest accuracy and AUC (0.8917 and 0.9626, respectively), followed by deep bite (0.8586 and 0.9238, respectively) and normal overbite (0.7591 and 0.8359, respectively). However, VSDs showed a different pattern between accuracy and AUC. Although the accuracy was highest for hypodivergent pattern (0.9094), followed by hyperdivergent pattern (0.9061) and normodivergent pattern (0.8309), the AUC was highest for hyperdivergent pattern (0.9730), followed by hypodivergent pattern (0.9684) and normodivergent pattern (0.9157).
In APSDs and VSDs, the total accuracy reached nearly 0.9 and the total AUC exceeded 0.95. However, VDDs showed a relatively lower total accuracy (0.8365 vs. 0.8880 and 0.8821) and total AUC (0.9074 vs. 0.9524 and 0.9523) than APSDs and VSDs.
Comparison of AUC values between internal and external test sets in binary ROC analysis (Table 4)
In APSDs and VSDs, Class III and open bite showed the highest AUC compared to other classifications (0.9807 and 0.9903 in the internal test set, 0.9475 and 0.9626 in external test set, respectively). However, VSDs showed a different pattern. The internal test set showed the highest AUC for hypodivergent pattern (0.9824), while the external test set showed the highest AUC for hyperdivergent pattern (0.9730). However, the difference in the AUC values was less than 0.01.
Comparison of AUC values between internal and external test sets in multiple ROC analysis (Table 5)
In terms of pairwise AUCs in the internal and external test sets of APSDs, VSDs, and VDDs, Class II vs. Class III ([II→III, 0.9913; II←III, 0.9920]; [II→III, 0.9992; II←III, 0.9989]; Δ value [II→III, 0.0079; II←III, 0.0069]), hyperdivergent pattern vs. hypodivergent pattern ([hyper→hypo, 0.9998; hyper←hypo, 0.9998]; [hyper→hypo, 0.9930; hyper←hypo, 0.9977]; Δ value [hyper→hypo, –0.0068; hyper←hypo, –0.0021]), and open bite vs. deep bite ([open→deep, 0.9982; open←deep, 0.9951]; [open→deep, 0.9924; open←deep, 0.9956]; Δ value [open→deep, –0.0058; open←deep, 0.0005]) showed the highest values in both the internal and external test sets and the smallest differences compared to other pairwise classifications.
t-SNE of APSDs, VSDs, and VDDs per dataset (Figure 5)
The GT in the training set, internal test set, and external test set showed that dots with different colors were mixed irregularly in the classification cutoff areas (dotted circle in Figure 5, GT) between the normal group (Class I in APSDs, normodivergent pattern in VSDs, and normal overbite in VDDs) and the other two groups (Class II and III for APSDs, hyperdivergent and hypodivergent patterns for VSDs, and open bite and deep bite for VDDs).
However, in the AI PD, the areas with irregular mixing had almost disappeared enough to indicate a cutoff line between the normal group and the other two groups in the training set, internal test set, and external test set (Figure 5, PD). This indicated that our model succeeded in creating good separation between the three classification groups in each diagnosis, resulting in consistent classification within each group.
Grad-CAM for each diagnosis (Figure 6)
Heat maps show differences in the location and size of the focus areas between three classification groups in each diagnosis. These indicate that our model can effectively use the information in the lateral cephalogram images.
DISCUSSION
The present study has some meaningful outcomes as follows: (1) Despite the different quality of lateral cephalogram images from diverse conditions of cephalometric radiograph systems in nationwide 10 hospitals (Table 1), a clinically acceptable accuracy of diagnosis was obtained for APSDs, VSDs, and VDDs; and (2) since it was possible to give a proper diagnosis for APSDs, VSDs, and VDDs with input of lateral cephalograms only, our model showed the possibility of general-purpose one-step orthodontic diagnosis tool.
Clinical meaning of the comparison results between internal and external test sets in binary and multiple ROC analysis
Comparison of accuracy with a previous study using binary ROC analysis results
Compared to model I of Yu et al.,8 our model showed slightly lower scores for total accuracy (< 0.011) and slightly higher scores for total AUC (< 0.020) (Table 6). Although our dataset had some disadvantages including a relatively smaller number of images in the dataset and an imbalanced data set compared to Yu et al.’s study8 (n = 5,890 lateral cephalogram images, and even distribution of data set after under-sampling), our model exhibited nearly the same performance as model I by Yu et al.8 To overcome this disadvantageous environment, we elaborated on constructing the proper architecture of our model using GN, ArcFace, and a softmax layer (Figure 2).
Excluding specific data, especially in the test set, may increase the risk of sample selection bias and lead to inaccurate validation of the model. Therefore, in the present study, all datasets with a whole distribution were included to properly validate the model (Figure 3).
Difference in the AUC values of in Class II and Class III groups in APSDs and hyperdivergent and hypodivergent groups in VSDs in binary and multiple ROC analysis
The hypodivergent group showed a higher AUC score than the hyperdivergent group in the internal test set, while the hyperdivergent group showed a higher AUC than the hypodivergent group in the external test set (0.9824 vs. 0.9730 in the internal test set, respectively; 0.9684 vs. 0.9730 in the external test set, respectively; Table 4).
The Class III group showed higher AUC values than the Class II group, which was in accordance with the results of Yu et al.8 for both internal and external test sets (0.9807 vs. 0.9533 in the internal test set, respectively; 0.9930 vs. 0.9601 in the external test set, respectively; Table 4). The reason might be a difference in the location and size of the focus areas in the diagnosis of VSDs and APSDs (i.e., relatively larger difference between Class II and Class III groups compared to between the hyperdivergent and hypodivergent groups; Figure 6). Further studies are necessary to investigate the reason why the Class III group showed a higher AUC than the Class II group.
Lower AUC values in VDDs compared to APSDs and VSDs in binary ROC analysis
The lower AUCs in VDDs in both internal and external test sets (Table 4) and relatively unclear separation of the normal overbite group from the deep bite and open bite groups in the GT of the t-SNE result (Figure 5) might be due to two reasons: (1) the imbalanced data composition in the training set, internal test set, and external test set (normal overbite, 61.3%, 52.9% and 43.6%; open bite, 26.5%, 29.7% and 28.2%; deep bite, 12.2%, 17.4% and 28.2%, respectively; Table 3) or (2) an inherent problem in the superimposed image between the anterior teeth.
Current status of CNN-based orthodontic diagnosis
Most previous CNN studies have focused on detecting cephalometric landmarks and/or calculating cephalometric variables for a two-step automated diagnosis.1-3,8-11 The study design, methods, and results of previous CNN studies are summarized in Table 7. In the present study, we proposed a one-step orthodontic diagnosis model, which only needs input of lateral cephalograms. The degree of performance of the AI model used in this study was comparable to the human gold standard (Tables 4 and 5). Automated AI-assisted procedures might save clinicians valuable time and labor in classification of skeletodental characteristics in a large sample size. However, it still needs an ultimate decision from a human expert, especially in borderline cases.
Limitations of this study and suggestions for future studies
The present study has some limitations. First, this study had a relative imbalance in the data sets of some centers. Second, more demographic, clinical, and cephalometric parameters should be included in setting the gold standard and training AI models in future studies.
As suggestions for future studies, it is necessary to develop a one-step automated classification algorithm for diagnosis of transverse and asymmetry problems. Prospective studies with larger diagnostic cohort data sets will allow more robust validation of the model.
CONCLUSION
The accuracy of our model was well-validated with internal test sets from two hospitals as well as external test sets from eight other hospitals without issues regarding the continuity of the data sets or exaggerated accuracy.
Our model shows the possible usefulness of a one-step automated orthodontic diagnosis tool for classifying skeletal and dental discrepancies with input of lateral cephalograms only in an end-to-end manner. However, it still needs technical improvement in terms of classifying VDDs.
XML Download