During resident programs, residents are required to develop knowledge, technical skills, patient communication skills, and advanced social skills to facilitate interactions with co-workers1). The United States Orthopaedic In-Training Examination (US OITE) was carried out in 1963 by the American Academy of Orthopaedics234567891011). In-training examination is an important method of assessing the knowledge of residents, determine minimal knowledge standards in training programs and develop appropriate programs for residents preparing for the Korean Board of Orthopaedic Surgery certification examination. In-training examinations for various orthopedic topics have been reviewed and analyzed. By evaluating and comparing the results, medical residents can analyze their strengths and weaknesses in medical knowledge, and hospitals can utilize these results to evaluate their curricula and develop better alternatives712). Previous studies of the US OITE classify questions by general topic, question type or reference, compare residents' performance by grading, and analyze the correlation between the US OITE and the actual examination6813). The Korean Orthopaedic In-Training Examination (KOITE), produced by the Korean Orthopedic Association (KOA), was first performed in 1980. Each year, a sitting the KOITE, a score sheet is provided to all residents through faculty members in their hospital. This allows faculty members to compare resident education and knowledge as well as to find study source for the Korean Board of Orthopaedic Surgery certification examination. However, insufficient analyses of the KOITE have been performed, and no study has focused on the hip and pelvis section. Therefore, this study aimed to analyze the hip and pelvis section of the KOITE to identify patterns in question content, recommended references and facilitate development of appropriate teaching programs.

The authors evaluated the KOITE during a 5-year period (2010 to 2014). A total of 500 questions were examined, 75 of which were in the hip and pelvis section (Fig. 1). The contents were analyzed and categorized according to topic, imaging modality, taxonomic classification, and references. Topics were classified as trauma (including fracture and dislocation), disease, anatomy, and basics (hip joint arthroplasty, hip joint biomechanics, and physical examination). Sub-topics were classified as biomechanics, anatomy, physical examination, fracture/dislocation, periprosthetic fracture of the femur, arthroplasty, arthroscopy, pelvic osteotomy, osteonecrosis of the femoral head, Legg-Calvé-Perthes disease (LCP), slipped capital femoral epiphysis (SCFE), femoro-acetabular impingement (FAI), piriformis syndrome, transient osteoporosis of the femur, etc. Most sub-topics were classified based on the diagnosis.

The questions were also divided according to taxonomic classification as described by Buckwalter et al.14) and Frassica et al15). Simple-recall or knowledge questions (taxonomy 1) involve recollection of facts without specific cognitive problem-solving. Diagnosis-type questions (taxonomy 2) involve deduction using given information such as clinical history and radiologic examination results. Evaluation and decision-making questions (taxonomy 3) require the examinee to establish a diagnosis and treatment plan using the information provided in the question. Diagnostic tools included imaging modalities, physical examinations, and laboratory tests. The imaging modalities were simple radiographs, magnetic resonance imaging/magnetic resonance arthrography, computed tomography (CT)/CT arthrography, and clinical photographs.

Also, the references of each question are investigated. The standard of reference is determined by whether it is possible to infer the answer based on the contents of the reference. Since the KOA publishes the KOA Textbook of Orthopaedics (7th edition), Campbell's Operative Orthopaedics (12th edition), Rockwood and Green's Fractures in Adults and Children (7th edition), the American Academy of Orthopaedic Surgeons Instructional Course Lecture (ICL of AAOS), and the Journal of Korean Orthopaedic Association (JKOA) and Clinics in Orthopaedic Surgery (CiOS) journal as reference books for medical residents, this study used the same standard. The ICL of AAOS, JKOA and CiOS were published from 2010 to 2014.

Finally, the total average score of the examination and the average score of the hip and pelvis section were obtained using the score sheet, and the results were analyzed in terms of their difficulty.

The overall weight of the hip and pelvis section of the KOITE was 15.0% (75 of 500 questions) for 5 years (Fig. 1). The most frequently asked questions were about fracture and dislocation (26/75, 34.7%), followed by basics (23/75, 30.7%), disease (21/75, 28.0%), and anatomy (5/75, 6.7%) (Fig. 2). A greater number of questions focus on disease. In terms of sub-topics, questions related to fracture and dislocation (25/75, 33.3%) and arthroplasty (20/75, 26.7%) were the most commonly asked, and every year at least one question covered anatomy, periprosthetic fracture of the femur, osteonecrosis of the femoral head, and arthroscopy (Table 1).

Basic descriptive questions were the most frequently asked (63/75, 84.0%), and questions including visual materials (such as figures, graphs, and videos) were asked at a relatively constant rate (Fig. 3A). In terms of visual material type questions, simple radiographs were provided most commonly, (6/12, 50.0%) followed by videos, clinical photographs, and CT images (Fig. 3B).

The most common types of question were about an appropriate treatment plan (taxonomy 3, 39/75, 52.0%), followed by simple knowledge (taxonomy 1, 29/75, 38.7%) and diagnosis (taxonomy 2, 7/75, 9.3%). Of the 46 questions regarding appropriate diagnosis and treatment, the largest proportion provided a basic description without a figure (38/46, 82.6%), among which 20 questions described a physical examination (20/38, 52.6%) and 4 physical examination and laboratory results (4/38, 10.5%). Of the four questions, two addressed hip joint infection and the other two osteoporosis and venous thromboembolism.

The most commonly cited reference was Campbell's Operative Orthopaedics (52/75, 69.3%), followed by the KOA Textbook of Orthopaedics (16/75, 21.3%), Rockwood and Green's Fractures in Adults and Children, and the JKOA and CiOS (Table 2).

The total average score of the KOITE of second to fourth grade residents was nearly 50 points between 2010 and 2014, and the degree of difficulty remained constant. Also, the average score of the hip and pelvis section of the KOITE maintained similarity with the total average score, but showed higher average scores in 2012 and 2013, unlike the overall trend (Fig. 4).

The KOITE began in 1980, but few analytical or descriptive studies have been conducted. Two papers focused on the shoulder and elbow section and the hand and wrist section1617). As faculty members and orthopedic surgeons at teaching hospitals in the hip and pelvis surgery subspecialty, our interest in training of orthopedic residents and the role of the KOITE led us to perform this study. This is the first study to analyze the questions of the hip and pelvis section of the KOITE during a 5-year period (2010 to 2014).

The examination comprised total of 500 questions from 2010 to 2014, which were divided into 11 domains: hand, shoulder and elbow, spine, hip, infection and metabolism, foot, trauma, pediatrics, tumor, basics and rehabilitation. Of these, the hip and pelvis section comprised 75 questions (15.0%). According to previous reports, 5.8% of the questions during the same period were in the shoulder and elbow section and 12.2% in the hand and wrist section1617). Although further analysis of other anatomic regions is required, the hip and pelvis section occupies a relatively large proportion. Furthermore, questions relating to the hip and pelvis section are asked every year at a relatively constant rate (12-18%).

In terms of topic, questions related to trauma including fracture and dislocation occupied the largest proportion (26/75, 34.7%), as reported by Kim et al.17) for the hand and wrist section of the KOITE. However, unlike the hand and wrist section, questions about basics (23/75, 30.7%) were as frequent as those regarding trauma. Among the questions regarding basics, those related to arthroplasty represented the largest proportion (20/75, 26.7%). Therefore, arthroplasty is one of the most important components of further training, together with fracture and dislocation.

Regarding question sub-types, anatomy, periprosthetic fracture of the femur, arthroscopy, and osteonecrosis of the femoral head were asked almost every year due to their importance. Moreover, all of the four questions regarding periprosthetic fracture of the femur dealt with fracture management after total hip arthroplasty, and from 2011 to 2014 the five questions about osteonecrosis of the femoral head dealt with treatment, with the exception of one question in 2010, which dealt with diagnosis. Therefore, an understanding of the appropriate diagnosis, classification and treatment plan of these two subtopics is required.

Of the questions, 84% were of the simple descriptive type, as reported previously for the hand and wrist section17). Furthermore, taxonomy 2 and 3, which inquire about appropriate diagnosis and treatment, were most frequently asked using descriptive text, without images or videos (82.6%). These types of question seem inappropriate for current medical circumstances, which require a comprehensive method of diagnosis and treatment that takes into account the results of examination and medical imaging. Therefore, such questions seem inadequate for evaluating the residents' critical thinking, and a question format that nurtures inferential thinking should be developed. The total average KOITE scores from 2010 to 2014 were around 50 points, suggesting that the difficulty of the examination was constant. Also, the average score of the hip and pelvis section was around 50%, similar to the total average score, but the average scores in 2012 and 2013 were higher than the total average scores in those years. The examination in 2012 comprised only basic descriptive-type questions, which lacked visual materials. Furthermore, the examination in 2013 included a larger proportion of basic descriptive-type questions compared to those in other years. Although the other parts of the KOITE were not analyzed, the data suggest that the average score of the hip and pelvis section increases with an increasing proportion of basic descriptive-type questions in the examination.

The most frequently cited references were Campbell's Operative Orthopaedics (69.3%) and the KOA Textbook of Orthopaedics (21.3%), which indicates that enough questions are not based on the JKOA and CiOS despite the KOA recommendation. According to previous reports, regular journal meetings can improve residents satisfaction with their education and increase the educational value of teaching hospitals13), and in the US OITE, 59% and 10% are from papers published within the last 5 and 1 years, respectively18). This indicates that frequent journal meetings can enable residents to acquire the latest knowledge, thereby achieving educational goals and maximizing performance. Therefore, there is a need to reference the Science Citation Index (SCI) and SCI Expanded (SCIE) journals in future examinations.

This is the first study to analyze the questions in the hip and pelvis section of the KOITE. There are several limitations associated with this study. First, the validity of the KOITE was not sufficiently evaluated because the examination was not analyzed according to residents' scores. A future study should involve analysis of not only residents' scores by grade but also the effect of changes in the questions on the residents' scores. Second, not all past examinations were analyzed comprehensively because only a 5-year period of the 35 years of the examination was analyzed. Third, comparison of the characteristics of the hip and pelvis section with other sections is problematic because of the lack of studies addressing the latter. Further studies should analyze the trends in each section, which will enable establishment of an appropriate training curriculum. Lastly, there was a change in the KOITE in 2015. The KOITE committee decided to use tablet PCs for the examination, to facilitate provision of images and videos. However, this study did not include post-2015 data. Therefore, the effect of use of tablet PCs on residents' scores and question types should be evaluated.

The dearth of analyses of the KOITE to date indicates that the effort to evaluate the propriety of training education and improve assessment of medical residents has been inadequate. This study provides information relating to the content and recommended references in the KOITE hip and pelvis section. This information will provide orthopedic trainees, orthopedic residency programs, and the Korean orthopedic board examination committee valuable information, which will improve resident knowledge and performance and optimize the hip and pelvis educational curriculum.