PDF
Citation
Print
INTRODUCTION
Temporomandibular joint disorders (TMD) represent a significant public health concern, affecting an estimated 5–12% of the general population. They refer to a group of conditions characterized by pain and loss of function and are associated with headaches, restricted mouth opening, hypermobility syndromes, fibromyalgia, anxiety, and sleep disorders.1-4 Given the impact of TMD on daily life, patients seek prompt solutions. In recent years, several innovations have emerged in various domains, including treatment modalities and patient management strategies for TMD. However, notable challenges remain for health professionals due to the complex nature of these diseases and their etiologies. Studies show that 50–80% of patients obtain information about their disease online before visiting a doctor.5
Artificial intelligence (AI) is a technology that can solve complex problems like human thinking and mimic human cognitive processes.6 In recent years, the development of large language models (LLMs) has revolutionized the field of AI.7,8 A multitude of chatbots such as Woebot, Your.MD, HealthTap, Cancer Chatbot, VitaminBot, Babylon Health, Safedrugbot, Microsoft Bing, and Google Bard are utilized for diverse purposes within the realm of health.9-11 In dentistry, AI is used in scheduling appointments, clinical diagnosis and treatment planning, malocclusion detection in orthodontics, automatic classification of restorations in panoramic radiographs, periodontal diseases, root caries, and detection of maxillofacial abnormalities.12
The most discussed and used chatbot is Chat Generative Pre-Trained Transformer (ChatGPT), developed by the San Francisco-based company OpenAI and released in November 2022. The chatbot was trained on a large dataset and functions in a manner analogous to that of a human in dialogue with users. ChatGPT uses AI to respond to natural language queries, such as those of humans.13-15 Its popularity is due to its detail, speed, and convenience. ChatGPT can provide numerous services to healthcare professionals in dentistry and healthcare.16-18 ChatGPT has been subjected to three examinations by the United States Medical Licensing Examination, and its capabilities have been demonstrated.19
In addition, it is important to note that AI models, particularly LLMs, can produce varying results depending on the phrasing of the input and quality of the datasets on which they were trained. These models may occasionally demonstrate inconsistencies in their responses and, in certain instances, generate false or incomplete information, a phenomenon known as hallucinations. Hallucinations are more prevalent when a model lacks sufficient or consistent data on a given topic, particularly in cases involving obscure or controversial subjects.20,21
Several articles on AI and LLMs have recently been published in various fields. 8-19 These publications evaluated the competence and reliability of ChatGPT, focusing on the views of health professionals. Despite a comprehensive literature review, no studies have been conducted on TMD. This study evaluated the reliability and usefulness of the ChatGPT-4 responses to TMD keywords. This study assessed ChatGPT’s efficacy in informing patients and professionals about TMD, a growing concern, and its reliability and utility.
We propose the following hypothesis: ChatGPT, with its extensive training on multiple topics, provides reliable and useful information for patients with TMD. This study is an effective supplementary resource for understanding TMD symptoms, management options, and self-care strategies, increasing patient knowledge and improving patient outcomes.
MATERIALS AND METHODS
This study was conducted following the principles of the Declaration of Helsinki. Ethical committee approval was not required because no human or animal data were used in this study.
Subheadings with various etiologies have been identified for TMD. The most recent guideline from the American Academy of Orofacial Pain and the International Headache Society, as modified by Okeson, was used as keywords.22 These are “masticatory muscle disorders,” “disc-condyle complex irregularities,” “structural disorders of the articular surfaces,” “inflammatory disorders of temporomandibular joint (TMJ),” “chronic mandibular hypermobility,” “ankyloses,” and “growth disorders of TMJ.” A dialogue was initiated with the AI service (ChatGPT-4) utilizing the entry “TMD.” A detailed discussion of the specific etiology of ChatGPT has been conducted. To obtain comprehensive information, we first asked ChatGPT about the disease. We then asked about causes, symptoms, and treatment. We created a new account to ensure that ChatGPT provides impartial answers. Each keyword was initiated as a new conversation and recorded for analysis. ChatGPT responses were obtained from the version released on October 12. The wording of the ChatGPT questions was standardized and consistent. Responses were examined for inconsistencies and hallucinations (Supplementary data).
The content of each response was evaluated for reliability using the ChatGPT reliability score.14 The ChatGPT reliability score is a Likert-type scale with scores ranging from 1 to 7. Responses from medical and scientific sources were verified to ensure they were free of incomplete or error-prone information. High scores indicate high usefulness (Table 1).
The ChatGPT usefulness score was used to assess the content of each response to determine its utility in patients.17 The scale is a Likert-type scale with values ranging from 1 to 7. A high score indicated high usefulness (Table 1).
The DISCERN scale was designed to evaluate the reliability of the ChatGPT responses. The modified DISCERN (mDISCERN) scale comprised five questions with yes/no answers (Table 1). According to the mDISCERN criteria, a score of 1 was negative and 5 was positive. Scores of 2, 3, and 4 indicated partial applicability. A score of “partially” is preferred for limited applicability. The total scores range from 0 to 5, with higher scores indicating greater reliability.
The Global Quality Scale (GQS) was modified with a specific emphasis on assessing the accuracy and utility of the information initially presented by Bernard. The responses generated by ChatGPT were evaluated using a GQS (Table 1).23
Two specialists of independent fields—an orthodontist and a physical medicine specialist—were involved in evaluating the screenshots to avoid bias. If the raters disagreed, a third independent rater assessed the data and made the final decision.
Statistical analysis
Analyses were performed using MedCalc® Statistical Software version 19.7.2 (MedCalc Software Ltd., Ostend, Belgium; https://www.medcalc.org). Descriptive statistics were used to describe the continuous variables (mean, standard deviation [SD], minimum, median, and maximum). Quantitative data were summarized as the mean, SD, and median (minimum to maximum). The Wilcoxon signed-rank test determined relationships between two dependent variables that did not conform to a normal distribution. The inter-rater agreement was analyzed using Cronbach’s α. The intraclass correlation coefficient results indicate that positive values ranging from 0 to 0.2 represent poor agreement, 0.2 to 0.4 represent fair agreement, 0.4 to 0.6 represent moderate agreement, 0.6 to 0.8 represent good agreement, and 0.8 to 1.0 represent very good agreement. The level of statistical significance was set at P < 0.05.
RESULTS
Two raters with extensive experience and independence evaluated the responses. The results are presented in the following tables and figure.
The responses are shown in Table 2, as indicated by a Likert-type reliability scale. The median Likert reliability scores were 6 (4–6) and 5 (4–6), respectively, indicating that the responses were highly reliable and reproducible for raters 1 and 2. The median reliability scores showed no statistically significant differences between the groups (P > 0.05; Table 3). The Inter-rater reliability of the scoring process was evaluated for the entire content. A high level of agreement was observed between the two raters (Cronbach α: 0.829).
Table 3 presents the distribution of the responses to the Likert-type usefulness scale. The median Likert usefulness score was 5 (4–6) for two raters, indicating that the responses were moderately helpful. The agreement of usefulness scoring between the two raters was examined for all content. Moderate agreement was observed between the two raters (Cronbach α: 0.693). The median usefulness scores showed no statistically significant differences between the groups (P > 0.05; Table 3).
The highest Likert scores for the reliability and usefulness of the information provided by the ChatGPT were for masticatory muscle disorders (mean: 6.0 ± 0). The lowest scores in terms of both reliability and usefulness were for inflammatory disorders of the TMJ (mean: 4.3 ± 0.6 for reliability, mean: 4.0 ± 0 for usefulness). The scores the two raters gave ranged between four and six (P = 1.000–0.317, respectively).
A comparative analysis was performed on all topics’ total reliability and usefulness scores. No statistically significant difference was found in any subject for the reliability or usefulness total scores (P = 0.083-1.000; Table 3).
Table 2 presents the distribution of ChatGPT response scores according to the mDISCERN scale. No statistically significant difference was identified between the median mDISCERN values of the groups (P > 0.05). Regarding reliability, as determined using the mDISCERN tool, Rater 1 identified very high reliability; however, Rater 2 identified high reliability (mean ± SD: 4.1 ± 0.7, 3.8 ± 0.6, respectively). The median mDISCERN score of the responses was 4 (3–5) for two raters. A statistically significant difference was observed in the mean mDISCERN scores between raters 1 and 2 (P = 0.046; Table 3). There was no statistically significant correlation between DISCERN and GQS scores (r = –0.006, P = 0.980).
The mean GQS scores for rater 1 and rater 2 were 3.8 ± 0.8 and 4.0 ± 0.5, respectively. The raters assessed the responses as of moderate to high quality. As demonstrated in Table 3, the GQS scores indicate a low level of agreement between the two raters (Cronbach α: 0.452). No statistically significant differences were observed between the groups (P > 0.05).
The reliability and usefulness indices were calculated based on the mean scores assigned by the raters. The resulting distribution is shown in Figure 1. The highest mean value for reliability was observed in masticatory muscle disorders and ankylosis (mean ± SD: 6.0 ± 0). In contrast, the lowest mean value was noted in chronic mandibular hypermobility (mean ± SD: 4.0 ± 0), as determined by the separate scoring of two independent raters. For usefulness, the highest mean value was seen in masticatory muscle disorders and disc-condyle complex derangement (mean ± SD: 6.0 ± 0), whereas the lowest mean value was seen in chronic mandibular hypermobility (mean ± SD: 4.0 ± 0). The raters had no significant differences in the reliability or usefulness total scores.
DISCUSSION
Recently, AI has brought about innovative tools such as ChatGPT, which are designed to engage in conversations and provide information on various topics. ChatGPT can be used in various areas of dentistry, such as symptom assessment and temporary treatment suggestions, appointment scheduling and reminders, the planning and assessment of treatments, the analysis and diagnostics of images, the monitoring of patients and the subsequent follow-up, and the conducting of literature reviews.14,24
The effects of AI on dentistry span various clinical and administrative domains. Clinically, AI is primarily used for diagnostic purposes, enhancing the accuracy and efficiency of identifying oral conditions, such as dental caries and periodontal diseases.25 Moreover, AI-driven imaging technologies have advanced the detection capabilities of dental radiographs, enabling the early and accurate diagnosis of complex oral issues. AI contributes to practice management by streamlining processes, such as appointment scheduling, patient communication, and claims processing, allowing dental professionals to devote more time to direct patient care.26,27 Innovations like AI-powered live coaching systems and computer vision technologies for pediatric and orthodontic care are anticipated to improve patient outcomes and practice efficiency.28
Despite these benefits, the integration of AI into dentistry is challenging. Ethical concerns, data privacy issues, and potential errors are the prominent controversies surrounding the adoption of AI in clinical practice.29
In addition, the quality of training data heavily influences the efficacy of AI tools, necessitating continuous updates. These challenges highlight the importance of adopting a cautious approach emphasizing the transparency and reliability of AI applications. As AI technology matures, its potential to transform dental-care delivery and patient engagement will continue to grow, fostering a new era of precision and enhanced patient-centered care.30 Health-related misinformation can lead to misdiagnosis, treatment errors, and serious consequences.31
Given the above scientific data, we aimed to assess whether ChatGPT displays a high level of knowledge on specific topics. To this end, we examined the reliability and usefulness of the information provided by ChatGPT concerning the various subtopics related to TMD affecting the daily lives of patients.
This study had several strengths. The authors employed a combination of objective and validated tools to assess reliability. These included Likert scales, GQS, and mDISCERN criteria, mainly compared with previously published descriptive studies.32,33 The Likert scale is an effective tool for evaluating the reliability and usefulness of generated responses, as it allows for a clear distinction between the levels of accuracy. This scale enables discernible differentiation among the levels of accuracy and is commonly used in the existing literature for comparable assessments.34-36
Discern is a validated instrument recognized as an effective method for assessing the reliability and quality of written health information. It is intended for health professionals and laypeople (Institute of Health Sciences, University of Oxford, Oxford, UK).37 In the present study, as suggested by previous studies, the mDISCERN scale was used to assess the reliability of the information to focus on specific criteria through a detailed analysis of TMD.38
TMJ problems require multidisciplinary treatment involving the cooperation of many specialists. Therefore, our study evaluated the responses separately by an orthodontist and a physical therapy practitioner. This can provide valuable insights into the ChatGPT platform’s strengths and limitations in addressing the diverse aspects of TMJ problems and help ensure a more reliable and holistic evaluation.
Our results indicate that masticatory muscle diseases exhibited the highest reliability and usefulness scores. The responses to masticatory muscle disorder treatment were evaluated, with a mean score of 6.0 ± 0. After reviewing the responses, a wide range of information was provided regarding treatment options for masticatory muscle diseases in the literature. These options vary from conservative treatments such as occlusal splints and medicines to more interventional approaches such as Botox injections and surgical procedures.39 ChatGPT’s treatment options were comprehensive and accurate in the context of masticatory muscle disorders. However, other studies require additional details. Accurate information on conservative treatments should be highlighted. However, superficial information on advanced surgical options is a potential limitation. The model’s information was incomplete and lacked guidance in complex areas like TMJ. This is a limitation of ChatGPT. The lowest mean usefulness and reliability scores were associated with inflammatory TMJ diseases. The reasons for this lower score could vary, including the complexity of the subject matter, quality of the available information, and presentation of the content.
However, ChatGPT can provide helpful information, and it is vital to highlight the significance of seeking guidance from qualified healthcare experts and reliable medical sources when making decisions regarding TMD. It is necessary to consider ChatGPT as an additional source of information. However, it does not replace the expertise of actual doctors. Previous research noted that ChatGPT frequently recommended contacting an orthodontist, dentist, or doctor at the end of responses. This result is consistent with our findings.31
Although the raters agreed on the usefulness and reality scores, they disagreed on the GQS or mDISCERN scores. Therefore, a third evaluator was included, and the scores were incompatible. The mDISCERN grading of the responses revealed that the level of agreement between the two evaluators is not particularly strong, and the observed agreement may be attributed to chance (Cronbach α: 0.452). In particular, there were differences in the scores related to patient treatment. This may be because the raters treated patients using different clinical practices.
Balel40 evaluated the answers generated by ChatGPT regarding maxillofacial surgery by physicians using the GQS scale; the results were of moderate quality. In their current study, Kılınç and Mansız41 used the Flesch-Kincaid and DISCERN tools to evaluate the reliability and readability of the information about orthodontics generated by ChatGPT-4. The mean DISCERN value was reported to be 2.96 ± 0.05 for general questions, 3.04 ± 0.06, 2.38 ± 0.27 and 2.82 ± 0.31 for treatment-related questions. The mDISCERN tool indicated high reliability for both raters (mean ± SD: 4.1 ± 0.7 to 3.8 ± 0.6, respectively). It is thought that in our study, the discrepancy in evaluations may be attributed to the different specialties of the evaluators.
Dursun and Bilici Geçer35 used the Likert scale, mDISCERN, GQS, and Flesch Reading Ease Score to assess ChatGPT-3.5, ChatGPT-4, Gemini, and Copilot AI. Responses were moderately reliable and of good quality. ChatGPT-4’s mean GQS score was 3.8 ± 0.62, indicating high quality. These results confirmed our findings. In our study, ChatGPT-4’s responses for TMD were moderate to high quality. The highest mean Likert score for the ChatGPT-4 responses was 4.5 ± 0.61. We found the mean Likert scores 5.4 ± 0.7 and 4.9 ± 0.8 for reliability and usefulness.
Tanaka et al.42 evaluated the reliability of ChatGPT-4 in answering questions on clear aligners, temporary anchorage devices, and digital imaging in orthodontics using a Likert scale. The study found a slight agreement and discrepancy in ratings between assessors, with a combined Fleiss’ κ value of 0.004 (P < 0.001). Our study demonstrated a high level of agreement regarding inter-rater reliability (Cronbach α: 0.829).
Khanagar et al.43 noted that in the context of AI applications for diagnosis and treatment planning in orthodontics, AI achieved accuracy and precision comparable to that of trained examiners. Abu Arqub et al.44 reported that ChatGPT generated responses to questions about clear aligner therapy in orthodontics with less-than-optimal accuracy. However, none of these studies referred to TMD.
Response variability and hallucination risk in AI models
LLM’s accuracy depends on the questions and training data despite their large datasets. Some inconsistencies are due to the model’s answers to the questions. The formulation of questions can result in discrepancies in model responses. Hallucinations, a prevalent issue in language models, must be considered. The generation of false or fabricated information by the model presents a significant risk, particularly in clinical applications.45
ChatGPT can generate coherent responses but may fabricate details or provide outdated information. A systematic review found that 96.7% of studies were concerned about the accuracy of ChatGPT. These concerns highlight the risks of ChatGPT, including misinformation, lack of originality, and hallucinations.46 ChatGPT’s effectiveness depends on the training data. It struggles with rare or novel medical conditions but shows potential in certain contexts, including clinical documents and healthcare efficiency. Additionally, studies have reported remarkable accuracy in specific medical examinations, suggesting that it could be useful in medical education and decision-making.47
In this study, some of the responses from the model were either incomplete or incorrect. ChatGPT responses to complex topics, such as inflammatory disorders of the TMJ, were incomplete or lacked detail. This demonstrates the need to rigorously evaluate language models when providing health information. Similarly, the aforementioned studies highlighted the phenomenon of hallucinations in ChatGPT responses.
Our study has several limitations. The ChatGPT 4.0 model can provide information on many subjects. ChatGPT 4.0 is based on an updated knowledge base until April 2023. However, they may not be aware of the recent developments. However, ChatGPT cannot verify or update the accuracy of the information provided. ChatGPT does not assess individual health or private medical histories. ChatGPT provides general information rather than personalized medical advice. To evaluate individual health conditions accurately, it is essential to consult qualified healthcare professionals. The absence of a published study on this topic prevented us from discussing it in detail.48
A previous study on TMD on YouTube found that information was inadequate. It is important to evaluate sources, as they may not provide accurate information, particularly for complex medical conditions.49 ChatGPT’s main advantage is its conversational interactivity.50 Studies are needed to compare the information provided by ChatGPT with YouTube and other patient information platforms. This may highlight areas of improvement and help patients make informed decisions.
CONCLUSIONS
Although AI has immense potential in healthcare, it should always be used with human expertise. Future research should focus on analyzing various LLMs. The proliferation of misinformation has become a major concern in the digital age. Academic institutions are important in ensuring that patients with TMD are directed toward the correct diagnostic and therapeutic processes. AI models have limitations, including variability in responses and the risk of hallucinations. However, the information provided by these models provide must be verified by experts in clinical applications.
XML Download