Journal List > J Korean Med Sci > v.39(33) > 1516088277

TOOLS
Yun, Ryu, Kim, Yoon, Choi, Park, Bae, Yim, and Myung: Subtyping of Performance Trajectory During Medical School, Medical Internship, and the First Year of Residency in Training Physicians: A Longitudinal Cohort Study
Original Article
Medicine General & Health Policy

Journal of Korean Medical Science 2024; 39(33): e239.

Published online: 19 July 2024

DOI: https://doi.org/10.3346/jkms.2024.39.e239

Subtyping of Performance Trajectory During Medical School, Medical Internship, and the First Year of Residency in Training Physicians: A Longitudinal Cohort Study

Je-Yeon Yun1, 2, *, Hyunjin Ryu3, *, Ju Whi Kim3, Hyun Bae Yoon3, Seung Choi4, Wan Beom Park3, 5, Eun Jung Bae6, 7, Jae-Joon Yim3, 5, Sun Jung Myung3

1Department of Neuropsychiatry, Seoul National University Hospital, Seoul, Korea.

2Yeongeon Student Support Center, Seoul National University College of Medicine, Seoul, Korea.

3Office of Medical Education, Seoul National University College of Medicine, Seoul, Korea.

4Department of Educational Psychology, College of Education, University of Texas at Austin, TX, USA.

5Department of Internal Medicine, Seoul National University College of Medicine, Seoul, Korea.

6Department of Pediatrics, Seoul National University Children’s Hospital, Seoul, Korea.

7Department of Pediatrics, Seoul National University School of Medicine, Seoul, Korea.

Address for Correspondence: Sun Jung Myung, MD, PhD. Office of Medical Education, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul 03080, Korea. issac73@snu.ac.kr

Equal contributor:

*Je-Yeon Yun and Hyunjin Ryu contributed equally to this work as co-first authors.

Received 1 April 2024       Accepted 1 July 2024

© 2024 The Korean Academy of Medical Sciences.

The Korean Academy of Medical Sciences

https://creativecommons.org/licenses/by-nc/4.0/
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (https://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Developmental trajectories of clinical skills in training physicians vary among tasks and show interindividual differences. This study examined the predictors of medical internship performance and residency entrance and found subtypes of performance trajectory in training physicians.

Methods

This retrospective cohort study involved 888 training physicians who completed a medical internship between 2015 and 2019. After the internship, 627 physicians applied for residency training between 2016 and 2020. Finally, 160 of them completed their first-year residency in internal medicine, surgery, pediatrics, and psychiatry departments between 2016 and 2020. Pearson’s correlation coefficients of internship performance and first year-residency performance (n = 160) were calculated. Latent profile analysis identified performance trajectory subtypes according to medical school grade point average (GPA), internship performance, English proficiency, and residency selection procedures. Multivariate logistic regression models of residency acceptance (n = 627) and performance in the top 30%/lower 10% in the first year of residency were also constructed.

Results

Medical internship performance showed a significant positive correlation with the medical school GPA (r = 0.194) and the written score for the medical licensing examination (r = 0.125). Higher scores in the interview (adjusted odds ratio [aOR], 2.57) and written examination (aOR, 1.45) of residency selection procedures and higher medical internship performance (aOR, 1.19) were associated with a higher chance of residency acceptance. The latent profile analyses identified three training physician subgroups: average performance, consistently high performance (top 30%), and adaptation to changes (lowest 10%). Higher scores in the interview for residency selection (aOR, 1.35) and lower scores for medical internship performance (aOR, 0.79) were associated with a higher chance of performing in the top 30% or lowest 10% in the first year of residency, respectively.

Conclusion

Performance in the interview and medical internship predicted being among the top 30% and lowest 10% of performers in the first year of residency training, respectively. Individualized educational programs to enhance the prospect of trainees becoming high-functioning physicians are needed.

Graphical Abstract

jkms-39-e239-abf001.jpg

Keywords: Graduate Medical Education Training, Medical Internship, First Year of Medical Residency, Workplace-Based Assessment, Latent Profile Analysis, Multivariate Logistic Regression

INTRODUCTION

The transition from being a medical student to a training physician can be complex and stressful, especially given uncertainties regarding how to enter and pursue a postgraduate training program for a specific specialty.1 While training physicians might share similarities during residency training,2 the developmental trajectories of clinical skills could also vary among the different tasks (indicated by curvilinear slopes for history-taking, clinical reasoning, diagnostics, documentation, and presentation [linear slope for teamwork]), and there are also interindividual differences.3 Competence in practice-based learning follows a curvilinear growth curve in the early stages of residency training,4 and the accuracy of self-evaluation tends to improve in the later stages of residency training.5 Of note, approximately 10–15% of training physicians in residency programs may not achieve a sufficient level of proficiency for at least one sub-competency at any time during their residency.26 Positive predictors of success in the residency training of family medicine include having a higher level of clinical knowledge, professionalism, proficient communication skills, a positive attitude, the ability to accept and integrate feedback, and high emotional intelligence.7 Moreover, a lower likelihood of burnout in medical residents of internal medicine, surgery, and emergency medicine is associated with higher self-efficacy, professional development, meaningful work, and hospital support.8 To enhance the accuracy of predictions and aid preparation for residency applications and training, program directors and training physicians should strive to identify more dependable indicators of residency performance in both the undergraduate and postgraduate periods.
Among other factors, the level of academic achievement in medical school may be associated with that in residency training. Failing the American Board of Internal Medicine Certification Exam on the first attempt is associated with older age on entering residency among medical residents of internal medicine; a lower percentile rank on the Internal Medicine In-Training Examination in each of the first, second, and third years of training; and lower scores on United States Medical Licensing Examination (USMLE) Steps 1, 2 (Clinical Knowledge), and 3.9 In fact, the level of medical knowledge in medical residents measured via the in-training examination is positively correlated with the examination scores for USMLE Steps 1 and 2.6 Moreover, sufficient medical knowledge in residency training might be a prerequisite for professionalism as a certified physician. Previous studies on training physicians in the United States have shown that higher levels of medical knowledge reduce diagnostic errors and the likelihood of severe disciplinary actions, while also being associated with superior performance in leadership and other areas.10111213
However, little information is available regarding the various subtypes of performance growth curves among training physicians after graduation from medical school. Moreover, few studies have subtyped training physicians according to their performance trajectory. Therefore, in the current study, we subtyped training physicians in terms of their performance trajectory and examined predictors of residency entrance and performance during the internship and first-year residency. For this purpose, we determined to what extent performance in a medical internship, where medical interns work in multiple diverse departments for 1 year, has predictive power for future performance as a resident. We also explored whether the grade point average (GPA) during medical school can serve as a predictor of performance as a resident, considering its role as an indicator of cognitive function, conscientiousness, and self-management ability.

METHODS

Study design and participants

This retrospective cohort study involved 888 medical interns (Table 1) who completed an internship certified by the Accreditation Council of the Korean Hospital Association from 2015 to 2019 with monthly evaluation by program directors at Seoul National University Hospital (SNUH), a university-affiliated tertiary general hospital with 1,950 physicians and 1,782 beds (http://www.snuh.org). The interns’ mean age at the start of the medical internship was 26.8 ± 2.3 years, and the male:female ratio was 0.56:0.44. We collected various data regarding the interns, including their demographics, GPA during medical school, scores on the written medical licensing examination, and scores on language proficiency tests such as the Test of English for International Communication (TOEIC), Test of English as a Foreign Language (TOEFL), and Test of English Proficiency developed by Seoul National University (TEPS). Additionally, we obtained average performance scores during the medical internship, focusing on patient care, communication, clinical skills (including cardiopulmonary resuscitation), self-directed learning ability, patient safety, and professionalism. Medical internship performance was scored monthly by the training director of each department according to the trainee’s allocation in the monthly rotation schedule. The reliability of the score was assessed through annual determination of the Cronbach’s α coefficient, which ranged from 0.915 to 0.960.
Table 1

Demographic characteristics of training physicians who completed a medical internship between 2015 and 2019 (N = 888)

jkms-39-e239-i001
Year No. of internship No. of male:female No. of drop-outs
2015 186 98:88 4
2016 181 95:86 4
2017 180 90:90 0
2018 180 109:71 3
2019 182 121:61 11

Statistical analysis: prediction of medical internship performance

Among the 888 training physicians who completed a medical internship between 2015 and 2019 (Table 1), 627 applied for a residency training program at SNUH between 2016 and 2020 for specialties where there was competition for residency entrance during the study period (such as internal medicine, pediatrics, psychiatry, neurology, rehabilitation medicine, family medicine, ophthalmology, or orthopedic surgery). Pearson’s correlation coefficients were calculated to examine the associations between internship performance and performance in other areas including medical school GPA, written exams (focusing on proper clinical case management in the fields of internal medicine, general surgery, obstetrics and gynecology, and pediatrics), semi-structured interviews (assessing clinical competencies, interpersonal competencies, and self-management), English proficiency, and special achievements during medical school (n = 627; Table 2). The threshold for statistical significance was set at P < 0.05/5 (where 5 represents the number of medical intern selection criteria) (i.e., P < 0.01).
Table 2

Pearson correlation coefficients of medical internship performance with other performance metrics in training physicians who completed a medical internship between 2015 and 2019 and applied for a residency between 2016 and 2020 (N = 627)

jkms-39-e239-i002
Performances Written part of medical licensing exam GPA of medical schoola Written exam for residency selectionb Interview for residency selection English proficiency Additional pointc
GPA of medical schoola
r 0.523**
P < 0.001
Written exam for residency selectionb
r 0.267** 0.339**
P < 0.001 < 0.001
Interview for residency selection
r −0.085* −0.023 0.322**
P 0.011 0.496 < 0.001
English proficiency
r 0.002 0.058 0.150** 0.065
P 0.952 0.083 < 0.001 0.052
Additional pointc
r 0.036 0.080* 0.027 0.073* 0.052
P 0.281 0.016 0.419 0.029 0.118
Medical internship performanced
r 0.125** 0.194** 0.058 −0.036 0.062 0.079*
P < 0.001 < 0.001 0.083 0.278 0.066 0.019
GPA = grade point average.
aGPA at medical school graduation.
bWritten test score for internship applicant selection.
cAdditional point for applicant with a special achievement, such as research output or a career in community service.
dInternship performance score for 1 year.
*P < 0.05; **P < 0.01.

Statistical analysis: prediction of acceptance in residency program

A receiver operating characteristic (ROC) curve was drawn, and the area under the curve (AUC) was derived to evaluate the diagnostic ability of each of these six items in discriminating between the accepted and unaccepted applicants (Fig. 1). For three items that showed high performance according to the ROC curves, the adjusted odds ratio (aOR) of acceptance into the residency was calculated with adjustment for other variables including sex, application year, GPA during medical school, TOEIC/TOEFL/TEPS scores, medical internship performance score, and scores for the residency selection procedures (written examination, practical test, and interview) (upper rows in Table 3).
Fig. 1

Receiver operating curves predicting residency acceptance in 627 training physicians who completed a medical internship between 2015 and 2019 and applied for the medical residency program between 2016 and 2020 based on medical school GPA, medical internship performance, English proficiency tests (TOEIC/TOEFL/TEPS), and written exam/practical exam/interview for residency selection.

GPA = grade point average, TOEIC = Test of English for International Communication, TOEFL = Test of English as a Foreign Language, TEPS = Test of English Proficiency developed by Seoul National University.
jkms-39-e239-g001
Table 3

Multivariate logistic regression analysis of residency acceptance and first-year residency performance

jkms-39-e239-i003
Dependent variables Outcomes Explanatory variables aOR (95% CI) P value
Residency acceptance in training physicians who completed a medical internship and applied for medical residency program between 2016 and 2020 (n = 627). Adjusted for sex, year, medical school GPA, TOEIC/TOEFL/TEPS scores, medical internship performance, written exam/practical test/interview for residency selection. Pass Medical internship performance score 1.19 (1.05–1.33) 0.005
Pass Written exam for residency selection (internal medicine, general surgery, obstetrics and gynecology, pediatrics) 1.45 (1.22–1.72) < 0.001
Pass Interview for residency selection 2.57 (2.17–3.04) < 0.001
First-year residency performance in training physicians who completed the first year of residency between 2016 and 2020 (n = 211; 160 of these training physicians also completed a medical internship between 2015 and 2019). Adjusted for medical school GPA, TOEIC/TOEFL/TEPS scores, medical internship performance, written exam/practical test/interview for residency selection. Top 30% Interview for residency selection 1.35 (1.03–1.76) 0.027
Lower 30% Medical internship performance score 0.76 (0.64–0.90) 0.001
Lowest 10% Medical internship performance score 0.79 (0.66–0.95) 0.013
aOR = adjusted odds ratio, CI = confidence interval, GPA = grade point average, TOEIC = Test of English for International Communication, TOEFL = Test of English as a Foreign Language, TEPS = Test of English Proficiency developed by Seoul National University.

Statistical analysis: subgrouping of trainees and prediction of first-year residency performance

For 211 training physicians (including 160 who completed their medical internship from 2015 to 2019) who completed their first year of residency training in internal medicine, surgery, pediatrics, and psychiatry departments at SNUH between 2016 and 2020, the average residency performance score was obtained. Residency performance in the first year of residency in terms of patient care, communication, clinical skills, self-directed learning ability, patient safety, system-based practice, and professionalism was scored monthly by the training director of each department according to the trainee’s position in the monthly rotation schedule. The reliability of this score was assessed through annual determination of the Cronbach’s α coefficient, which ranged from 0.891 to 0.960. Multivariate logistic regression analyses were performed to examine the utility of using medical school GPA, English proficiency test scores, medical internship performance, and written exams, practical exams, and semi-structured interviews for predicting the average performance in the first year of residency, classified as being within the top 30%, lower 30%, or lowest 10% (lower rows in Table 3). Further, for items with the highest explanatory power for the top 30%, lower 30%, or lowest 10% performance class in each department, we calculated the aORs of falling into these strata in the first year of residency. This calculation involved adjusting for other variables, including medical school GPA, medical internship performance score, English proficiency test scores, and scores from residency selection procedures (written examination, practical test, and interview).
Of note, for a subgroup of 160 training physicians who completed both a medical internship between 2015 and 2019 and their first year of residency between 2016 and 2020 at SNUH, we performed a latent profile analysis14 to identify subgroups with differential performance trajectories (Fig. 2A), including the medical school GPA, score for the written part of the medical licensing examination, medical internship performance score, English proficiency test scores, scores for the residency selection procedures, and first-year residency performance score. A latent profile analysis is a branch of Gaussian finite mixture modeling; it models the probability of each case belonging to a profile and identifies distinct subgroups within the data. In this study, the R package “mclust”15 was used to perform the latent profile analysis based on the z-transformed mean and standard deviation scores for each of the eight above-mentioned variables. Additionally, Pearson’s correlation coefficients among these eight variables were calculated (n = 160) (Fig. 2B). Further, for each separate department, i.e., internal medicine (n = 66, blue dots), surgery (n = 25, pink dots), pediatrics (n = 46, yellow dots), and psychiatry (n = 23, purple dots), scatterplots with marginal histograms (Fig. 3) illustrated the distributions and associations of the performance score in the first year of residency (y-axis) with the medical school GPA, written examination score in the medical licensing examination, medical internship performance, and residency selection procedures of the written examination (focusing on the proper clinical case management in the subjects of internal medicine, general surgery, obstetrics and gynecology, and pediatrics) and semi-structured interviews (assessing clinical competencies, interpersonal competencies, and self-management).
Fig. 2

Subtypes of performance trajectory in training physicians. (A) Latent profile analysis divided 160 training physicians who completed both a medical internship between 2015 and 2019 and the first year of residency training between 2016 and 2020 into three subgroups: average (n = 106), consistently high achiever (n = 37), and adapting to challenges (n = 17). (B) Correlation matrix (Pearson’s correlation coefficients are shown and are marked with red rims when P < 0.05; n = 160) among the zGPA, zLICENSE, zINTERN, zENG, scores for residency selection procedures including the written examination (zIGOP), zINTERVIEW, and zPRACTICE, and zR1raw.

zGPA = z-score–transformed medical school GPA, zLICENSE = z-score–transformed medical licensing examination score, zINTERN = z-score–transformed medical internship performance score, zENG = z-score–transformed TOEIC/TOEFL/TEPS scores, zIGOP = z-score–transformed internal medicine, general surgery, obstetrics and gynecology, and pediatrics, zINTERVIEW = z-score–transformed interview, zPRACTICE = z-score–transformed practical test, zR1raw = z-score–transformed first-year residency performance score.
jkms-39-e239-g002
Fig. 3

Associations between average residency performance in the first year of residency and (A) zGPA, (B) zLICENSE, and (C) zINTERN, as well as the (D) written examination (zIGOP) and (E) zINTERVIEW during the residency selection procedures (all x-axis), in 160 training physicians who completed both a medical internship between 2015 and 2019 and their first year of residency training between 2016 and 2020 (n = 66 from internal medicine [blue], n = 25 from general surgery [red], n = 46 from pediatrics [yellow], and n = 23 from psychiatry [purple]).

zGPA = z-score–transformed medical school GPA, zLICENSE = z-score–transformed medical licensing examination score, zINTERN = z-score–transformed medical internship performance score, zIGOP = z-score–transformed internal medicine, general surgery, obstetrics and gynecology, and pediatrics, zINTERVIEW = z-score–transformed interview.
jkms-39-e239-g003
The Cronbach’s alpha coefficient was calculated to determine the reliability of the performance score of the medical internship and first year of residency. Statistical analyses were performed using SPSS software (version 23.0; IBM Corp., Armonk, NY, USA).

Ethics statement

The SNUH Institutional Review Board (IRB) reviewed and approved protocol of this study (IRB No. 2007-019-1139). The requirement for written informed consent was waived because of the study’s retrospective cohort design and use of anonymous data. All study procedures were performed in accordance with the principles stated in the Declaration of Helsinki.

RESULTS

Predictors of medical internship performance

Table 2 shows Pearson’s correlation coefficients among the medical internship performance score, medical school GPA, score in the written part of the medical licensing examination, English proficiency test scores, and interview during medical internship selection (n = 888; training physicians who completed a medical internship between 2015 and 2019). Notably, the medical internship performance score showed a significant positive correlation with the medical school GPA (r = 0.194, P < 0.001) and score for the written part of the medical licensing examination (r = 0.125, P < 0.001). On the contrary, scores for TOEIC/TOEFL and residency selection procedures did not have significant associations with internship performance (all P > 0.05).

Predictors of acceptance into residency program

Fig. 1 shows the AUC values derived from the ROC curves of training physicians who completed their internship and applied for the medical residency program at SNUH from 2016 to 2020 (n = 627). The residency selection interview (AUC = 0.899), residency selection practical test (AUC = 0.766), and medical internship performance score (AUC = 0.739) showed good discrimination ability (AUC of > 0.7 for all) between accepted and unaccepted training physicians. The aORs derived from the multivariate logistic regression models (Table 3) demonstrated higher odds of acceptance into the residency program with higher scores in the interview (aOR, 2.57) and written examination (aOR, 1.45) of the residency selection procedures, in addition to the medical internship performance score (aOR, 1.19), after adjustment for sex, residency application year, medical school GPA, TOEIC/TOEFL scores, medical internship performance score, and residency selection procedures (written examination, practical test, and interview).

Subgrouping of training physicians and prediction of first-year residency performance

In the subgroup of training physicians who completed both a medical internship between 2015 and 2019 and their first year of residency in internal medicine, surgery, pediatrics, and psychiatry departments between 2016 and 2019 at SNUH (n = 160), the latent profile analysis identified subgroups with differential performance trajectories (Fig. 2A). Among the models of subgroups 1–5, the lower Bayesian information criterion in the 3-, 4-, and 5-class models than in the 1- and 2-class models suggested that the 3-, 4-, and 5-class models (Bayesian information criterion = −3,487.147, −3,510.343, and −3,510.434, respectively) fit the data better. The Lo–Mendell–Rubin test and bootstrap likelihood ratio test of the 3-, 4-, and 5-class models indicated that the 4-class model was more effective than the 3-class model (P = 0.001) and comparable to the 5-class model (P = 0.055). To ensure adequate participant representation in each subgroup (n > 10), we ultimately selected the 3-class model. This model included subgroups representing average performance (n = 106), consistently high achievers (n = 37, top 30%), and those adapting to challenges (n = 17, lowest 10%).
Considering all of these 160 training physicians combined, performance in the first year of residency was significantly associated with the scores of both medical internship performance (r = 0.179, P < 0.05) and the written part of the medical licensing examination (r = 0.176, P < 0.05) (Figs. 2B, 3B, and C) but not with the medical school GPA or residency selection procedures (Figs. 2B, 3A, B, and E; all P > 0.05). On the contrary, and supporting the suitability of the subgroups of performance trajectory distinguished in this study (Fig. 2A), the aOR derived from the multivariate logistic regression models (lower rows in Table 3) showed that training physicians who completed their first year of residency training between 2016 and 2020 at SNUH (n = 211), regardless of whether they completed a medical internship between 2015 and 2019 at SNUH, had higher odds of achieving performance in the top 30% if they had higher scores in the residency selection interview (aOR, 1.35); a lower medical internship performance score was associated with increased odds of performance within the lowest 10% in the first year of residency (aOR, 0.79).

DISCUSSION

To the best of our knowledge, this study is the first to subtype the performance trajectory of training physicians across medical school, medical internship, and the first year of residency training. In this study, training physicians who completed their medical internship and first year of residency (n = 160) could be categorized into three subtypes (Fig. 2A): average performance (n = 106), consistently high performance (n = 37, top 30%), and adaptation to changes (n = 17, lowest 10%). This finding is consistent with previous studies of medical students. During the final 2 years of preclinical education, medical students demonstrate heterogeneity of learning trajectories and coursework performance, which are associated with their Medical School Admission Test scores and USMLE Step 1 scores.16 Upon medical school graduation, medical students could be divided into higher achievement and moderate achievement subgroups based on their self-assessed core professional skills.17 Further, during residency, training physicians need to reach the practicing member stage, where they enhance their independent clinical and team management skills, refine their decision-making abilities, and develop team leadership skills.18 However, the diversity of both individual and workplace-related factors might affect the developmental trajectory of physician performance. First, distinctions arise among training physician subgroups in terms of prioritizing work-related factors within the team, the acknowledgment of individual efforts in the workplace, and proficiency in stress management skills, all of which contribute to global well-being.19 Second, there is potential discordance in the evaluation of clinical skills when comparing self- and supervisor assessments,20 possibly hindering the feedback-related clinical development of training physicians. Third, learning environments differ according to the teaching facilities of the hospital, the average teaching performance, and the percentage of time spent on educational activities by the faculty in each department.19 Therefore, there is a need for postgraduate medical educational programs that are specifically tailored to the diverse trajectories of performance. Such programs should aim not only to promote excellence but also to address and rectify the specific shortcomings identified in training physicians.21
We also found a positive correlation between medical internship performance and first-year medical residency performance (r = 0.179; Fig. 2). In other words, a lower medical internship performance score was associated with increased odds of exhibiting performance within the lowest 10% during the first year of residency training (aOR, 0.79; Table 3). Because of the low amount of variance attributed to the learner (< 10%), in addition to the effects of confounders such as the developmental stage of faculty members as educators and raters,22 scale selection, and perceived differences in faculty grading stringency,23 a larger number of observations (median of 60) might be required for reasonable reliability of workplace-based assessments based on the Core Entrustable Professional Activities for Entering Residency.1722 Collectively, our results highlight the potential significance of averaging medical internship performance scores across 1 year of monthly evaluations within each department. This approach may be valuable for achieving a comprehensive understanding of clinical performance across diverse working conditions.
Regarding the factors that could potentially explain the variance in medical internship performance among training physicians, we found a positive correlation between medical internship performance and medical school GPA. This is in accordance with previous studies that have predicted academic performance based on training performance. In one study, medical residents with higher performance in medical school along with higher USMLE Step 1 and 2 scores demonstrated significantly higher residency performance scores in four domains (clinical, surgical, academic, and global performance) and were more likely to pass the board examination on their first attempt.24 In another study, higher USMLE Step 2 Clinical Knowledge scores were associated with lower odds of overcoming performance-related difficulties arising during residency.25 These findings align with the hypothesis put forth by program directors in medical schools, suggesting that core clerkship performance serves as a reliable indicator of an applicant’s readiness for residency. This is particularly evident in Accreditation Council for Graduate Medical Education domains such as Medical Knowledge, Patient Care, and Procedural Skills.26
The current study also demonstrated higher odds of achieving top 30% performance in the first year of residency among training physicians who scored high in the residency selection interview (aOR, 1.35). This aligns with earlier research emphasizing the predictive value of structured interviews, where standardized job-related questions applied uniformly to all applicants were able to forecast residents’ performance, including in-training examination scores,27 clinical2829 and academic29 performance, placement in the top third in the final evaluation of the residency,30 successful completion of the residency,29 and pursuit of academic jobs following residency.27 A 1-point increase in the overall interview score given by general faculty during residency selection correlated with reduced odds of encountering performance-related challenges during residency.25 However, training physicians with “quiet,” “shy,” or “reserved” behavioral traits might be overlooked in terms of both interviews and residency performance compared to their more extroverted peers.31 With support from natural language processing in deciphering the semi-structured interview,32 more comprehensive assessment of core values linked to resident success—such as intellectual curiosity, compassion, communication, work ethic, teamwork, self-awareness, professionalism, and adaptability33—could be achieved. This enhanced evaluation would be applicable to both the residency selection interview and the assessment of residency performance.
This study had three main limitations. First, it involved medical interns affiliated to only a single institution. Additional studies encompassing multiple training hospitals worldwide are required to confirm the generalizability of the current study findings. Second, possible variability in factors related to the performance of the training physicians among the diverse medical specialties and different stages of residency was not examined. Future studies involving sufficient numbers of training physicians engaged in diverse medical specialties would help to address this issue. Third, this study had no information regarding the self-efficacy or subjective well-being of the training physicians. If the viewpoints of the training directors and trainee themselves are simultaneously considered, a deeper understanding and more accurate prediction of training physicians’ performance trajectories might be possible.
This study examined predictors of medical internship performance and residency entrance and identified subtypes of performance trajectory in training physicians. Training physicians who completed the first year of residency were divided into average, consistently high achiever, and adapting to challenge subgroups. Medical internship performance was associated with medical school GPA. Acceptance into a residency was related to medical internship performance in addition to the residency selection procedures of interviews and written exams (internal medicine, general surgery, obstetrics and gynecology, and pediatrics). Both medical internship performance and interviews served as predictors for achieving top 30% or lowest 10% performance in the first year of medical residency. In light of these findings, there is a need for educational programs tailored to these potential predictors, aiming to achieve excellence and address the identified shortcomings in training physicians.21

Notes

Funding: This work was supported by the Seoul National University Hospital Research Fund (to S.J.M., grant No. 03-2020-0230) and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (to J.Y.Y., grant No. 2023R1A2C1006783).

Disclosure: The authors have no potential conflicts of interest to disclose.

Author Contributions:

  • Conceptualization: Yun JY, Ryu H, Myung SJ.

  • Data curation: Yun JY, Ryu H, Myung SJ.

  • Formal analysis: Yun JY, Ryu H, Myung SJ.

  • Methodology: Yun JY, Ryu H, Myung SJ.

  • Software: Yun JY, Ryu H, Myung SJ.

  • Validation: Yun JY, Ryu H, Myung SJ.

  • Investigation: Yun JY, Ryu H, Myung SJ.

  • Writing - original draft: Yun JY, Ryu H, Myung SJ.

  • Writing - review & editing: Yun JY, Ryu H, Kim JW, Yoon HB, Choi S, Park WB, Bae EJ, Yim JJ, Myung SJ.

References

1. Farah G, Gonzalez MH, Choubey AS, Karam M, Igram C, Dougherty P, et al. Be a lighthouse for your medical students: how to help them navigate during changing times to a successful match. Instr Course Lect. 2024; 73:87–95. PMID: 38090889.
2. Tanaka P, Park YS, Roby J, Ahn K, Kakazu C, Udani A, et al. Milestone learning trajectories of residents at five anesthesiology residency programs. Teach Learn Med. 2021; 33(3):304–313. PMID: 33327788.
3. Colbert-Getz JM, Lappe K, Gerstenberger J, Milne CK, Raaum S. Capturing growth curves of medical students’ clinical skills performance. Clin Teach. 2023; 20(6):e13623. PMID: 37605795.
4. Rutgers DR, van Raamt F, Ten Cate TJ. Development of competence in volumetric image interpretation in radiology residents. BMC Med Educ. 2019; 19(1):122. PMID: 31046749.
5. Quick JA, Kudav V, Doty J, Crane M, Bukoski AD, Bennett BJ, et al. Surgical resident technical skill self-evaluation: increased precision with training progression. J Surg Res. 2017; 218:144–149. PMID: 28985841.
6. Cassidy DJ, Chakraborty S, Panda N, McKinley SK, Mansur A, Hamdi I, et al. The surgical knowledge “Growth Curve”: predicting ABSITE scores and identifying “At-Risk” residents. J Surg Educ. 2021; 78(1):50–59. PMID: 32694087.
7. Nazerali-Maitland A, Douglas C. Challenges with international medical graduate selection: finding positive attributes predictive of success in family medicine residency. BMC Prim Care. 2022; 23(1):256. PMID: 36175829.
8. Ju TR, Mikrut EE, Spinelli A, Romain AM, Brondolo E, Sundaram V, et al. Factors associated with burnout among resident physicians responding to the COVID-19 pandemic: a 2-month longitudinal observation study. Int J Environ Res Public Health. 2022; 19(15):9714. PMID: 35955071.
9. Seaberg PH, Kling JM, Klanderman MC, Mead-Harvey C, Williams KE, Labonte HR, et al. Resident factors associated with American board of internal medicine certification exam failure. Med Educ Online. 2023; 28(1):2152162. PMID: 36443907.
10. Mamede S, Zandbergen A, de Carvalho-Filho MA, Choi G, Goeijenbier M, van Ginkel J, et al. Role of knowledge and reasoning processes as predictors of resident physicians’ susceptibility to anchoring bias in diagnostic reasoning: a randomised controlled experiment. BMJ Qual Saf. 2024.
11. Jones AT, Brethauer SA, Dent DL, Desai DM, Jeyarajah R, Barry CL, et al. How does the sequence of the American board of surgery examinations impact pass/fail outcomes? Ann Surg. 2024; 279(1):187–190. PMID: 37470170.
12. Reisdorff EJ, Johnston MM, Kraus CK, Keim SM, Santen SA. Association between the American board of emergency medicine oral certifying examination and future state medical board disciplinary actions. J Am Coll Emerg Physicians Open. 2024; 5(1):e13119. PMID: 38322376.
13. Govindarajan V, Shah AH, Morell AA, Borowy V, Ingle SM, Mendez Valdez MJ, et al. Predicting academic career placement via development of novel intra-residency metrics. World Neurosurg. 2023; 174:e35–e43. PMID: 36841537.
14. Spurk D, Hirschi A, Wang M, Valero D, Kauffeld S. Latent profile analysis: a review and “how to” guide of its application within vocational behavior research. J Vocat Behav. 2020; 120:103445.
15. Scrucca L, Fop M, Murphy TB, Raftery AE. mclust 5: clustering, classification and density estimation using gaussian finite mixture models. R J. 2016; 8(1):289–317. PMID: 27818791.
16. Song X, Jia Y. Using latent class growth analysis to detect group developmental trajectories in preclinical medical education. Adv Health Sci Educ Theory Pract. 2024; 29(3):803–812. PMID: 37679596.
17. Grbic D, Gielissen KA, Obeso V, Amiel JM, Jayas A, Andriole DA. Core entrustable professional activities for entering residency: a national survey of graduating medical students’ self-assessed skills by specialty. J Am Coll Surg. 2022; 235(6):940–951. PMID: 36102502.
18. Herzog TL, Sawatsky AP, Kelm DJ, Nelson DR, Park JG, Niven AS. The resident learning journey in the medical intensive care unit. ATS Sch. 2023; 4(2):177–190. PMID: 37533538.
19. Silkens ME, Chahine S, Lombarts KM, Arah OA. From good to excellent: improving clinical departments’ learning climate in residency training. Med Teach. 2018; 40(3):237–243. PMID: 29172795.
20. Sorenson JM, Khan NR, Michael LM 2nd, Nguyen V, Baughman B, Boop FA, et al. Case curve: a novel web-based platform and mobile phone application to evaluate surgical competence in graduate medical education. Neurosurgery. 2024; 94(6):1237–1245.
21. Nguyen ET, Berler MH, Gonzales PA, Greenberg AL, Lebares CC, Divino C, et al. Flourishing and the prioritization of workplace elements in general surgery residents. J Surg Res. 2023; 291:488–495. PMID: 37536190.
22. Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, et al. How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study. Med Educ. 2024; 58(7):825–837. PMID: 38167833.
23. Zavodnick J, Doroshow J, Rosenberg S, Banks J, Leiby BE, Mingioni N. Hawks and doves: perceptions and reality of faculty evaluations. J Med Educ Curric Dev. 2023; 10:23821205231197079. PMID: 37692558.
24. Tooley AA, Law J, Lelli GJ, Sun G, Godfrey KJ, Tran AQ, et al. Predictors of ophthalmology resident performance from medical student application materials. J Surg Educ. 2024; 81(1):151–160. PMID: 38036387.
25. Hemrajani R, Vettese T, Law K, Turbow S. Association of interview and holistic review metrics with resident performance-related difficulties in an internal medicine residency program. J Grad Med Educ. 2023; 15(5):564–571. PMID: 37781425.
26. Wang A, Karunungan KL, Story JD, Shlobin NA, Woo J, Ha EL, et al. Reimagining a pass/fail clinical core clerkship: a US residency program director survey and meta-analysis. BMC Med Educ. 2023; 23(1):788. PMID: 37875929.
27. Chen F, Arora H, Martinelli SM, Teeter E, Mayer D, Zvara DA, et al. The predictive value of pre-recruitment achievement on resident performance in anesthesiology. J Clin Anesth. 2017; 39:139–144. PMID: 28494890.
28. Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ. 2007; 64(6):378–385. PMID: 18063273.
29. Alterman DM, Jones TM, Heidel RE, Daley BJ, Goldman MH. The predictive value of general surgery application data for future resident performance. J Surg Educ. 2011; 68(6):513–518. PMID: 22000538.
30. Bhat R, Takenaka K, Levine B, Goyal N, Garg M, Visconti A, et al. Predictors of a top performer during emergency medicine residency. J Emerg Med. 2015; 49(4):505–512. PMID: 26242925.
31. Quinn JK, Mongelluzzo J, Nip A, Graterol J, Chen EH. Perception of quiet students in emergency medicine: an exploration of narratives in the standardized letter of evaluation. West J Emerg Med. 2023; 24(4):728–731. PMID: 37527382.
32. Drum B, Shi J, Peterson B, Lamb S, Hurdle JF, Gradick C. Using natural language processing and machine learning to identify internal medicine-pediatrics residency values in applications. Acad Med. 2023; 98(11):1278–1282. PMID: 37506388.
33. Drum B, Lamb S, Gradick C. Values-based resident selection in an internal medicine-pediatrics residency program. J Gen Intern Med. 2023; 38(6):1410–1416. PMID: 36344647.
TOOLS
Similar articles