The core components of medical education are knowledge, attitudes, and skills. Accordingly, the importance of practical evaluation in medical education has been emphasized [1]. To evaluate the clinical competence of medical students, some objective evaluation tools, such as the clinical performance examination (CPX) and objective structured clinical examination (OSCE), have been used [2]. The CPX evaluates whether history-taking, physical examination, and patient education have been performed correctly and whether desired patient–physician interactions have been achieved in the course of interviewing a standardized patient (SP) [3]. In the OSCE, the performance appropriateness of basic clinical procedures is evaluated based on an SP or via a model [3]. In South Korea, a clinical skills examination (CSE) has been implemented since 2010 as part of the Korean Medical Licensing Examination [4]. The CSE was introduced for the first time for Korean Dental Licensing Examinations in Korea in 2021 [5]. During the last decade, the practical ability of oriental medicine doctors has been emphasized [6]. As oriental medicine includes various unique medical techniques such as pulse examination and acupuncture, an emphasis has been placed on strengthening the practice-based competence of oriental medicine doctors [6]. According to a recent scoping review, there were 25 studies on the OSCE and CPX vis-à-vis oriental medicine in South Korea between 2011 and 2022, and this number had increased significantly over the last 3 years (i.e., studies from 2020 to 2022 accounted for 52% of the total) [7].

The Delphi method is used when there is no existing basis in relation to a statistical model, knowledge in the field in question is incomplete, and expert judgment is considered important [8]. It has been used widely in the health science and medical education fields [9]. Notably, oriental medicine doctors are very similar to traditional Chinese medicine doctors in China and Taiwan, although they differ in their knowledge systems and level of medical practice guaranteed by their license. Thus, there is no empirical evidence for reference in the field of studies on the Korean Oriental Medicine Licensing Examination. Therefore, the validity of introducing the CSE to the Korean Oriental Medicine Licensing Examination, which requires investigation, may be useful to address this gap in the literature.

The purpose of this study was to develop implementation strategies for the CSE for the Korean Oriental Medicine Licensing Examination. The detailed objectives included the following 3 questions: (1) What are the appropriate clinical presentations for the CSE? (2) What are the appropriate basic clinical skills for the CSE? (3) When and how should the CSE be introduced?

This study complied with the Declaration of Helsinki. It was approved by the Institutional Review Board of Gachon University Gil Korean Medical Hospital (GIRB-22-108) on September 5, 2022. Informed consent was obtained from all the study participants.

This Delphi study was completed electronically between July and November 2022.

A 3-round Delphi study following the recommended guidelines for the conducting and reporting of Delphi studies was undertaken [10].

To set the problem area, the authors conducted 4 investigations for clarification: a comprehensive literature review, an email-based survey of oriental medicine doctors registered with the Association of Korean Medicine (response rate: 9.28%; 2,221/23,946) and of professors at 12 oriental medicine schools (response rate: 41.4%; 206/500), and an advisory board meeting with the participation of the Association of Korean Medicine Colleges. The research methods and results for setting the problem area are described in Supplement 1.

The expert panel was set up based on official recommendations from related official organizations, including oriental medicine universities and the official subdivision of the oriental medicine society. A panel of 21 experts was formed, including 12 oriental medicine education experts and 9 subdivision representatives.

The working group comprised 8 researchers. All the working group members were oriental medicine doctors, and 7 of them were professors at oriental medicine schools. The principal researcher (H.Y.L.) has a doctoral degree in medical education.

The Delphi method was implemented over 3 rounds. Each round was conducted as an email-based survey to a panel of experts. Anonymity was guaranteed, and controlled feedback was provided. The objectives of the first, second, and third rounds were distinguished. In the first round (September 15–22, 2022), opinions were gathered from the expert panel on the list of potential candidates for the CPX and OSCE. The validity of introducing each item into the Korean Oriental Medicine Licensing Examination was measured with 6 (for basic clinical skills) and 7 (for clinical presentations) questions, including “necessity,” “feasibility,” and the RUMBA mnemonic (i.e., “relevant,” “understandable,” “measurable,” “behavioral,” and “achievable”). The questions were to be answered on a 3- or 5-point Likert scale. As there is no agreed mode of application of the Likert scale for this question, the use of 3 or 5 points was decided through consensus among the authors. Open-ended questions enabled the free expression of opinions vis-à-vis clinical presentations or basic clinical skills. Four subcommittees were formed by requesting official recommendations from related official organizations in connection to pulse examination, acupuncture, chuna, and sasang constitution to discuss the comments raised in the open-ended questions. These 4 topics are the hallmark clinical areas of oriental medicine in Korea. The subcommittees also discussed how to successfully implement these topics into the CSE for the Korean Oriental Medicine Licensing Examination. In the second (September 27–October 6, 2023) and third (October 22–November 3, 2023) rounds, the validity of detailed methods for the CSE was investigated. The third round included questions on which no consensus was reached in the second round. In this round, options other than the most frequent one according to the professor’s response could also be selected. The results of discussions in the subcommittees were provided, and the expert panel was asked to respond about whether or not it agreed with the aforesaid results.

The protocol of this study is presented in Supplement 2 and summarized in Fig. 1.

The average scores of the responses to the clinical presentations of the CPX and the list of basic clinical skills in the first round were calculated and ranked. Descriptive statistics were used to assess whether a consensus was reached in the second and third rounds. An agreement rate of 66.7% or higher was adopted as the definition of consensus.

The top 5 clinical presentations of the CPX included back pain (total average score=24.7), neck pain (24.4), omalgia (24.4), headache (24.2), and ankle pain (24.1) (Table 1). The top 5 basic clinical skills included cupping (total average score=19.3), moxibustion (19.0), acupuncture (19.0), medical records/medical certificate (18.9), and pharmacopuncture (18.9) (Table 2). The concerns raised from the open-ended questions during the first round were grouped into 4 clinical topics (Table 3).

A consensus was reached for 9 items in the second round, including items (b)–(j) (Table 4). In the second round, the 4 subcommittees discussed the concerns raised from the open-ended questions in the first round. The results of the discussion were summarized and presented in the third round. Items (d), (f), and (i) were included in the third round despite reaching a consensus, as some of the concerns raised warranted further discussion (Table 4).

No consensus was reached on item (a) in the second and third rounds (i.e., the year of introduction of the CSE). Consensus was reached for items (d), (f), and (i) in the second and third rounds. Among them, the consensus reached for items (d) and (f) differed from that in the second round: (d) the test time per station was 12 minutes (90.5%), and (f) the appropriate timing for the CSE was before the written test, and it should only be for a prospective graduate (66.7%). However, the same consensus was reached for item (i), namely, the CPX should be rated by 2 SPs (1 for acting and 1 for scoring) (71.4%) (Table 4). The feasibility evaluated through the subcommittees’ discussion on concerns raised was 3 points or higher on average for both relevant and measurable factors (Table 5). As this study was preliminary, the authors did not plan further rounds to arrive at a consensus.

This Delphi study investigated 2 topics: a list of clinical presentations and basic clinical skills that could be used and/or should be developed if the CSE were to be introduced into the Korean Oriental Medicine Licensing Examination and a detailed strategy for its implementation. Although the number of OSCE or CPX studies focusing on oriental medicine doctors has recently increased, educational resources for the CSE for oriental medicine doctors still need to be developed [7]. This study listed potential candidates’ clinical presentations and basic clinical skills for the introduction of the CSE into the Korean Oriental Medicine Licensing Examination through a comprehensive literature review and a survey of 2,221 oriental medicine doctors. Each item was evaluated by an expert panel, and our findings suggest tasks to prioritize for introducing the CSE to the Korean Oriental Medicine Licensing Examination. The priority lists of clinical presentations and basic clinical skills included “back pain,” “neck pain,” “omalgia,” “headache,” and “ankle pain” for clinical presentations (Table 1), as well as “cupping,” “moxibustion,” “acupuncture,” “medical records/medical certificate,” and “pharmacopuncture” for basic clinical skills (Table 2). The expert panel reached a partial consensus on the detailed implementation strategy in the second and third rounds. Notably, however, the consensus rate was low for the timing of the introduction of the CSE. It may be important to address the survey items for which a consensus was not reached, including the development of CSE resources and the formulation of detailed implementation strategies, in future research. Moreover, to establish a comprehensive evaluation framework, it is imperative to promptly initiate formal development procedures.

This study has several limitations. First, there are limitations with regard to the expert panel setting. This Delphi study established an expert panel based on official recommendations from related official organizations in South Korea. However, this does not guarantee that the participants were highly recognized education experts on oriental medicine education, especially the CSE. Nevertheless, as the importance of clinical skills training in oriental medicine education has only been emphasized relatively recently [7], and there are no clear criteria for the selection of such experts, the authors had no choice but to establish the expert panel with this limitation. Second, the expert panel’s opinions were not weighted. The lists of clinical presentations and basic clinical skills, which constituted the main components of the questionnaire in this study, might have been imbalanced owing to the expertise of the respondents based on clinical division. However, this study did not acknowledge differences in clinical expertise and gave equal weight to the opinions of the experts on all clinical presentations and basic clinical skills lists. Third, there was some discontinuity between the second and third rounds. Therefore, the stability of the responses could not be evaluated, and the possibility of a coincidence cannot be excluded from the consensus obtained. Finally, a Likert scale was used to evaluate items on necessity, feasibility, and RUMBA was evaluated for the clinical presentations and basic clinical skills. The importance was listed in order by summing the responses. However, as the legitimacy and weighting of the summation of these scales have not been verified, it is acknowledged that the methodology of this study is somewhat arbitrary.

Further research in this field may refer, but need not be limited, to the results of this preliminary study. First, basic research to introduce the CSE into the Korean Oriental Medicine Licensing Examination should be carried out. Studies should focus on areas such as developing a CSE implementation guide and simulation to evaluate the likelihood of the successful implementation of the guide. If necessary, a feasibility study on the introduction of the simulator can be conducted to facilitate the introduction of the CSE. Following this, a list of items to be used in the CSE should be officially developed, and the method of evaluation for each item should be established. Second, research on education and evaluation for the introduction of the CSE should be carried out. Research can be conducted to set a passing mark if the CSE is introduced to the Korean Oriental Medicine Licensing Examination and to develop educational textbooks for assessors and SPs. Finally, it would be desirable to conduct 1–3 simulation CSEs before the official introduction, and related laws and regulations must be revised for the official introduction of the CSE to the Korean Oriental Medicine Licensing Examination.

The main clinical presentations for the CSE were back pain, neck pain, omalgia, headache, and ankle pain. The essential basic clinical skills were cupping, moxibustion, acupuncture, medical records/medical certificate, and pharmacopuncture. A consensus on the implementation time of CSE into the Korean Oriental Medicine Licensing Examination was still not reached. This study is the first to propose the introduction of the CSE to the Korean Oriental Medicine Licensing Examination. It provides preliminary data that can be used as a reference while discussing the introduction of the CSE.