Licensing examinations in medicine are widespread across the world [1]. These examinations let us know that our doctors have achieved a minimum of knowledge and skills to provide quality care [2]. In addition, a higher score on a licensing examination may be associated with better access to residency programs, better work opportunities, and patient safety [3].

For Peruvian medical students, the licensing examination is called the National Licensing Exam (ENAM) [4]. Developed in 2003 by the Peruvian Society of Medical Schools, it remains the standard to ensure that doctors can practice medicine in Peru. The ENAM is a multiple-choice question exam, with a total of 180 questions mostly in the form of clinical cases based on the most important disease in Peru. Due to regulatory mechanisms and to increase its importance, the ENAM is now the biggest contributor to the selection of the Rural Service in Peru, and it has an influence on the selection of medical specialties in Peru [5,6].

In addition to its regulatory role, the ENAM informs the population on the level of knowledge of our future doctors. However, in 2021, a study conducted by Mendoza et al. alarmed medical educators, health policymakers, and the general population, mainly because they found a high rate (42.8%) of disapproval [7]. Although it was an important study, there was little medical education theory involved in the design and interpretation of results.

It is widely known that medical education research must be strongly based on theory, models, or a framework at the moment of design or analysis [8]. For early-career medical education researchers, theory may appear difficult and disturbing [9]; this phenomenon is well represented in the ENAM-related research and has led us to poor understanding of which factors influence ENAM scores.

Therefore, this review aimed to identify which associated factors have been studied with respect to ENAM scores in Peru, and to develop a framework to explain how these factors interact with the final score.

As this study was a systematic review, no human or human-origin materials were involved; thus, neither approval by the institutional review board nor the obtainment of informed consent is required.

This systematic review was described according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, available from: http://www.prisma-statement.org [10].

The inclusion criteria for this systematic review were: (1) studies published in English or Spanish; (2) studies published since 2003; (3) quantitative studies (analytical cross-sectional, case-control, cohorts, quasi-experimental, and experimental studies); and (4) studies that assessed the ENAM score as an outcome. Qualitative studies, and not fully accessible studies were excluded.

The search was conducted in September 2022 in 4 databases: MEDLINE, EMBASE, SciELO, and Web of Science. One register was searched: the National Register of Peruvian Research (RENATI, abbreviation in Spanish). To assess the gray literature, the first 10 pages of Google Scholar were assessed. Finally, all references of eligible studies were analyzed to identify further studies.

The search strategy was developed in PubMed (MEDLINE) and translated to other databases using a polyglot tool (https://sraccelerator.com/#/polyglot). The main search was:

#1: ENAM: ENAM or “Licensing Examination” OR “National Licensing Exam”

#2: Associated factors: correlation OR concordance OR differences OR association OR “associated factors”

#3: Peru: Peru OR Peruvian

#4: #1 AND #2 AND #3

For SciELO, Web of Science, RENATI and Google Scholar, only part #1 of the string was applied (Supplement 1).

The author (J.F.C.) eliminated duplicates using Zotero ver. 6.0 (Digital Scholar, Vienna, VA, USA). The remaining studies were assessed using the eligibility criteria to check the titles and abstracts. The full texts of eligible studies were evaluated, and the reference lists of these articles were reviewed to identify further studies.

The main author (J.F.C.) collected the data from selected studies using an extraction form prepared that included general data from studies, specific data using the academic achievement model by Alyahyan and Dustegor [11], and inferential data from studies.

As specified before, the data extracted were as follows: (1) general data from studies: first author, year, city of provenance, study objective, type of study (journal article, thesis, or other) and study design; (2) specific data: sample size, students’ sociodemographic characteristics (age, gender, nationality, marital status, and academic status), socioeconomic status, students’ environment (medical school, internship setting), learning activities (study resources, preparation time, others), psychological factors, measures of prior academic achievement (grade point average [GPA], other) and ENAM score; and (3) inferential data: variables with positive associations, variables with negative associations, variables with no associations, and regression models.

For measurements of psychological or learning activities the instruments were also extracted.

The main author (J.F.C.) evaluated studies using the Medical Education Research Study Quality Instrument (MERSQI). A tool that evaluates 6 domains (study design, sampling, type of data, validity of the evaluation instrument, data analysis, and outcomes), it is composed of 10 items and the score could range from 5 to 18 [12].

Data were tabulated in Microsoft Excel (Microsoft Corp., Redmond, WA, USA), then classified and analyzed to accomplish the main objective of this review. Each study was reviewed 3 times for a better classification of variables with positive associations, variables with negative associations, and variables with no association. The classification of data was conducted using the model of Alyahyan and Dustegor [11]; when in doubt about an instrument or variable, a senior researcher was approached. The extracted data were analyzed through a narrative synthesis, and then used to draw a figure to explain how all factors assessed interact with the ENAM score. Due to the heterogeneity and disparity among studies’ results, a meta-analysis was not performed.

To minimize reporting bias, the gray literature and RENATI were searched, mainly because most of the ENAM research was conducted as graduate dissertations or theses, and these are considered to be peer-reviewed by the dissertation committee.

Associated factors were reported as correlation coefficients (the Pearson correlation coefficient [r] or the Spearman rho [r_s]), and the following criteria were used to assess the relationships as strong (r=0.7 to 1), moderate (r=0.4 to 0.6) or weak (r=0.1 to 0.3), both for negative and positive relationships. Regarding regression models, data were extracted as reported (odds ratio, prevalence ratio, or risk ratio) and, if it was adjusted, the factors used for adjustment were noted.

Most of the studies were analytical and cross-sectional, with one case-control study. As established in the pyramid of evidence these studies are at the low end of the spectrum of evidence quality, but are those needed to answer the research question [13]. To assess the certainty of outcomes a P-value <0.05 with a confidence interval (CI) was the main criterion.

As shown in Fig. 1, 138 studies were identified using 4 electronic databases and one register, and 12 additional studies through the gray literature and citation searching. After duplicate removal, 129 studies were eligible for full-text review. Finally, 12 studies met the eligibility criteria for data extraction and analyses [7,14-24] (Table 1, Dataset 1).

Twelve identified studies included 38,500 students who took the ENAM. Most of the studies had an analytical cross-sectional design, with the exception of the study of Baldera Aquino and Alvarado Alva [15], which was a case-control study. The studies were published between 2011 to 2021, with most of them published between 2018 to 2021 (66.67%). Three studies were published in peer-reviewed scientific journals [7,14,18], while the remaining were published in RENATI as gray literature. The study samples ranged from 42 to 6,556 participants. The most evaluated factors were gender, GPA and age. The rate of disapproval of the ENAM ranged from 16% to 42.80%. The most frequently associated factors with the ENAM across studies were GPA, internship in EsSalud, older age (negative association) and regular academic status.

The minimum score within 6 of the domains is 1, and the maximum score across all domains is 3. Accordingly, MERSQI scores range from 5 to 18. As stated in Table 2 [7,14-24], the mean MERSQI score was 10.33 (range, 9 to 12.5). Three studies enrolled 3 and more institutions [7,17,18]. The data collected were objective, with the exception of one study where the ENAM score was self-reported [24]. Due to the nature of the studies, all outcomes across studies were knowledge in the Kirkpatrick framework of the MERSQI.

Different studies were included, not only through peer-reviewed journals, but also the gray literature from dissertations.

There is a high level of certainty for GPA with ENAM, due to its association across studies and because learning theories support this finding. Regarding other factors, there is a low level of certainty although P-values were <0.05. This uncertainty is for the following reasons: (1) heterogeneity among studies, (2) observational designs, (3) absence of theory used to design and interpret the studies, and (4) differences in size across studies.

This systematic review aimed to summarize the factors associated with ENAM scores among 12 studies. Although GPA remained the most studied factor, this review supports the fact that ENAM is a multifactorial outcome, as stated in previous models or reviews [11,25,26]. Therefore, in Fig. 2, a proposed model to understand how different factors interact with the ENAM score is presented.

The results of the present systematic review showed that the ENAM score, as well as that of other licensing examinations, is not a one-factor product, but a multifactorial effect. Although GPA was the most studied factor and nationality was the strongest predictor, other factors such as age, marital status, academic status, gender, simulation tests, study resources, preparation time, learning styles, study techniques, resilience, test anxiety, self-regulated learning strategies, medical school, and internship setting play a role in the final score (Fig. 2).

Previous studies conducted for the United States Medical Licensing Examination (USMLE) showed that previous academic achievement was correlated with higher USMLE scores [27]. A systematic review conducted in 2022 in the USMLE setting also found that step 1 scores, practice examinations, and GPA from high school, all indicators of previous academic achievement, were associated with higher step 2 scores [28]. Socioeconomic status plays an important role in education; as previously demonstrated by Jacobparayil et al. [28] and Giordano et al. [27], medical students with socioeconomic disadvantages and older age performed worst on USMLE examinations.

One finding that study resources do not have a major impact on ENAM scores contradicts the findings of a previous Best Evidence Medical Education review [29], according to which the use of an “off the shelf learning platform” was associated with higher scores on the USMLE, and associations were also found for the number of questions, reflection on mistakes, and rate of correct questions. This result differs from the findings of the present review, mainly because in the study of Sosa Espinoza [22], they only asked if students did or did not do practice questions.

Self-regulated learning was associated with higher ENAM scores, specifically the use of metacognitive, information seeking, and information processing strategies; these findings match those of Broadbent and Poon [30], according to whom self-regulated learning was associated with higher academic achievement. As a possible explanation for this finding, students with a higher use of metacognitive strategies can identify their mistakes on practice questions and adjust accordingly. Regarding test anxiety, the findings of this review are similar to those reported by Green et al. [31] in the USMLE setting, although in the same study test-taking strategies did not improve scores. However, a meta-analysis conducted on randomized controlled trials on higher education compared study skills training, behavior therapy, cognitive behavioral therapy, and combined therapies, showing that combined therapies had an estimated effect size of g=1.15 (95% CI, 0.33 to 1.96) on academic achievement [32].

One major limitation of this study is the higher rate of gray literature and heterogeneity in reports of desired outcomes, which made it unfeasible to conduct a meta-analysis. Another problem was the methodology and data collected among studies, in which few cofounders were analyzed.

A couple of years ago, one of my senior teachers told us that every medical student from Peru has the same opportunity to be a high achiever on the ENAM and it only depends on how “hard you study.”

By now, according to this review, I am glad to say that he was wrong, because the ENAM is a multifactorial phenomenon, and even if the GPA and nationality are the most important predictors, other factors are associated, such as academic achievement, student demographics, student environment, learning activities, and psychological factor, which had a role in the outcome. Therefore, the proposed model for the study of the ENAM score would help students, faculty, health policymakers, future researchers, and the public to better understand the ENAM.