There are numerous indexing services for journals, and most are not recognized by the government. Thus far, Science Citation Index Expanded (SCIE) and the Korean Citation Index (KCI) are currently recognized by the Korean government. Scopus and PubMed are used as additional evaluation tools, but they are not widely accepted in Korea. However, some European countries use the h-index to evaluate individual performance among researchers; these data are provided by Scopus. Accordingly, the influence of Scopus has been expanded. Publishing a manuscript in a high-impact journal may be an advantage for researchers with regard to grants and reputation1. Accordingly, indexed journals with high-impact factors will have more opportunity to publish higher quality articles. Regional journals may lose the opportunity to publish high-quality articles and to be accepted by major indexing services. When this trend is combined with the underestimation of specific fields, scientific advancement can be delayed.

Dentistry journals have been underestimated by Clarivate Analytics and Scopus for several decades in comparison with medicine journals. Oral surgery and dentistry journals have been placed in the same category by Clarivate Analytics2. This practice has also been used with the medicine category as it comprises several fields, including allergy, anatomy, anesthesiology, cardiovascular system, clinical neurology, dermatology, and emergency medicine2. Scopus is much better than Clarivate Analytics because it classifies seven different fields as part of the dentistry category3. However, Scopus also has many flaws in their classification system. For example, the oral surgery category comprises 59 titles, but many may not be classified as an oral surgery title. Of these 59 titles, some no longer publish under the same name, including Dental Cadmos, Mund-, Kiefer- und Gesichtschirurgie, Oral Surgery, and Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology, and Endodontics. The Journal of Maxillofacial and Oral Surgery was indexed to Scopus, but it is not anymore. Because the category dental implantology comprises multidisciplinary fields, it should be classified separately. If these titles are excluded from the oral surgery category, only 26 titles remain in the oral surgery category of Scopus.(Table 1) We also selected titles that do not fall into the oral surgery category of Scopus.(Table 2) SCIE places dentistry, oral surgery, and medicine collectively in the same field2. The total number of journals in this field is 90. Among them, only 10 titles may be classified as oral surgery titles.(Table 3)

If the titles are from the same category, there should be similarity among research fields. This similarity can be proved by analyzing references1. The hypothesis of this study was that the titles in the same category would have higher interactions. If some titles showed extremely low numbers of citations compared with other titles in the same category, the classification may be incorrect. The purpose of this article was to evaluate the classification system of major indexing services for journal titles in the oral surgery category by analyzing references, through which the relevance between the journal title and category could be determined.

Four different titles were selected arbitrarily for this analysis. The Journal of Oral and Maxillofacial Surgery (JOMS)45678910111213 was one journal title selected from journals indexed by Scopus and SCIE that was correctly classified as an oral surgery title.(Tables 1, 3) The Journal of the Korean Association of Oral and Maxillofacial Surgeons (JKAOMS)14151617181920212223 was one journal title selected from the journals indexed by Scopus only and was correctly classified as an oral surgery title.(Table 1) One journal title that was selected from journals not indexed by Scopus or SCIE and classified as an oral surgery title was Maxillofacial Plastic and Reconstructive Surgery (MPRS)24252627282930313233. One journal title that was selected from journals indexed by Scopus and SCIE and incorrectly classified as an oral surgery title by Scopus (Table 2) was The Journal of Prosthodontic Research (JPR)34353637383940414243.

The inclusion criteria were (1) being an original article or a review article and (2) having at least 30 references. Editorials, case reports, and technical reports were excluded. Articles were collected beginning on January 1, 2018. When 10 articles that met the criteria were collected, collection was finished. If any journal title that met the inclusion criteria in 2018 had less than 10 articles, the deficiencies were replenished by the articles published in 2017.

The references were classified into three categories (oral surgery, dentistry, and other). Among Scopus- or SCIE-indexed journals, classification as an oral surgery title was based on Tables 1 and 3. For unindexed titles, the terms “surgery” and “oral/maxillofacial” were used as the keywords for determining the classification category. Other titles in the dentistry category were classified as dentistry journals. Any title that was not classified as an oral surgery or dentistry title was classified as other.

After counting the total number of references in each article, the number of citations for each category was recorded. The ratio of citations among each category was calculated by dividing each value by the total number of references. The interaction between the three selected oral surgery titles and titles incorrectly classified by Scopus was analyzed.(Table 2) Additionally, the self-citation ratio was analyzed. Comparison of the four titles was done using ANOVA. For post hoc testing, Bonferroni's method was used. The level of significance was set at P<0.05. Statistical analysis was performed using IBM SPSS Statistics 23.0 (IBM Corp., Armonk, NY, USA).

The total number of references for each journal title was 463, 362, 411, and 459 for JKAOMS, MPRS, JOMS, and JPR, respectively. In the case of MPRS, two articles were collected from 2017. The rate of self-citation was 0.2%, 4.1%, 10.2%, and 2.0% for JKAOMS, MPRS, JOMS, and JPR, respectively. JOMS showed a relatively higher self-citation rate compared with the other three titles. When analyzing the citations in JKAOMS, MPRS, and JOMS from the titles listed in Table 2, the percentage of citations was 0.4% for JKAOMS, 4.4% for MPRS, and 0.2% for JOMS.

The percentage of citations from oral surgery journals was 26.7%±18.4% for JKAOMS, 24.5%±22.7% for MPRS, and 40.1%±28.0% for JOMS.(Fig. 1) However, the percentage of citations from oral surgery journals was only 1.1%±1.3% for JPR. This difference among groups was statistically significant (P<0.001). On post hoc test, JPR had a significantly lower value compared with that of JKAOMS and JOMS (P=0.046 and P=0.001, respectively).

The percentage of citations from dentistry journals excluding oral surgery journals was 11.9%±9.2% for JKAOMS, 34.4%±32.4% for MPRS, and 15.8%±14.0% for JOMS (Fig. 2), while the percentage of citations from dentistry journals excluding oral surgery journals was 80.6%±13.5% for JPR. The difference among groups was statistically significant (P<0.001). On post hoc test, JPR had a significantly higher value compared with the other journal titles (P<0.001).

Considering the increasing influence of major indexing services on the scientific community, extremely few studies have evaluated indexing services. To the best of our knowledge, this was the first research evaluating indexing services in the oral surgery category. According to the selected samples, the current oral surgery category was flawed in both Scopus and SCIE. The interaction pattern based on the reference analysis of oral surgery titles such as JKAOMS, MPRS, and JOMS was significantly different compared with dental journals such as JPR.(Fig. 1, 2) This preliminary study demonstrated that oral surgery journals had different patterns of interaction than dental journals and therefore should be classified as an independent category.

The percentage of citations from oral surgery journals was at least 20% in three oral surgery journals; however, it was approximately 1% for JPR.(Fig. 1) This is considered a major difference and should be considered an error in Scopus classification. Some authors claimed that a high percentage of mutual citations may be evidence of a citation cartel1. However, a journal article having a reasonable number of mutual citations should help determine the journal category. Arbitrary selection criteria were used in this study to classify a journal as an oral surgery journal; the criteria were having the word “surgery” and “oral/maxillofacial” in the title. In this study all three selected titles showed similar performance.(Fig. 1) Thus, the selection criteria for classifying a title as an oral surgery title in this study seemed functional.

In the case of MPRS, the percentage of citations from dentistry journals excluding oral surgery journals was higher compared with the other two titles.(Fig. 2) The reason for this could be that articles associated with dental implants showed higher interaction with dentistry journals, and not oral surgery journals. Yu et al.24 and Kakar et al.25 were associated with implant restorations 87% and 73% of their citations, respectively, were from dentistry journals excluding oral surgery journals. In addition, MPRS includes articles about maxillofacial application of tissue engineering.

The percentage of citations from dentistry journals excluding oral surgery journals was higher in JPR.(Fig. 2) In the case of orthodontic journals, some articles show high interaction with oral surgery journals. One article that was selected from The American Journal of Orthodontics and Dentofacial Orthopedics (AJODO) had 25 references, 10 of which are from oral surgery journals and eight of which are from dentistry journals excluding oral surgery journals44. However, another article45 selected from the same journal did not have any references from oral surgery journals, despite having 55 references. Because there was a high level of variance in topics, collecting big data will be necessary to draw reliable conclusions.

The limitation of this study was that the number of selected samples was small considering the vast number of publications in the field of oral surgery and dentistry. There were no definite title selection criteria for each category. In addition, there were no guidelines about how to manage multidisciplinary fields of oral surgery having close interactions with other dental categories such as dental implantology624 and orthognathic surgery44. In this article, any title including the word “implant” in was excluded from oral surgery titles. In addition, any title including the word “orthodontic” was also excluded from the oral surgery category. These exclusions are not relevant to the clinical practice of oral surgery specialists. If all oral surgery/dentistry titles were included using a cross-reference database, more confirmative results could be obtained. Because this study was a preliminary pilot study, subsequent confirmative studies are needed.

Analysis of selected samples demonstrated that some oral surgery titles had low interaction with other oral surgery titles. Therefore, the decision to include or exclude a journal from a particular category in Scopus and SCIE should be reviewed. If errors are discovered, revisions should be made accordingly. In addition, the scientific interaction of oral surgery titles may vary from that for other dental titles. Thus, oral surgery titles should be separated from dentistry titles and established as an independent category.