PDF
ePub
Citation
Print
Introduction
Around 10% to 15% of undergraduate medical students are likely to have substantial difficulties during their course [1]. The risk factors that have been associated with poor progress through medical college include failure in three or more examinations per year, an overall average score of less than 50%, health or social difficulties, failure to complete the hepatitis B vaccination on time and remarks noted about poor attitude or behavior [2]. At the medical college where I work, nearly a third of students fail in the first internal examination in anatomy. The remedial measures instituted for these students are stereotypical and include administration of written assignments and periodic tests. Despite these measures, 10% of students on an average fail in the summative university examinations. This is demoralizing and stigmatizing for the students and results in a loss of time and self-confidence. Extra classes need to be held for these students contributing to the additional burden on teachers.
Remediation is usually offered to medical students who underperform on examinations. A recent systematic review of the literature focused on why and how remediation measures might have succeeded [3]. The review included only those studies which had an intervention, provided retest data and had at least one outcome measure. Thirty-one studies which met the inclusion criteria were reviewed. Many of the studies were published after the year 2000, directed at medical students and aimed at improving their performance on subsequent examinations. Control or comparison groups, sample size estimations, theoretical frameworks and long-term follow-up were rare [3]. Though interventions were often complex, it was difficult to identify the active components that contributed to the success of the remedial measures [3].
It has been shown that the likelihood of successful remediation is increased if students at risk are identified early in the course. Encouraging an open and ongoing dialogue, systematic review and monitoring of progress achieved, studentcenteredness, individualization of remedial measures and use of appropriate resources with full faculty support have also been shown to improve outcomes from remediation [4]. Another study showed that low marks in the first semester examinations were the best single indicator of future academic performance [5].
Attempts have been made previously to use questionnaires to identify strugglers in a medical course [5]. Two cohorts of undergraduate medical students in Nottingham were administered the short-form General Health Questionnaire (GHQ-12) as a possible screening tool for future academic difficulties during the course. The results of the study showed that there was insufficient evidence to recommend the use of the questionnaire as a routine screening tool [5]. Another study conducted in Sri Lanka identified the non-cognitive and sociodemographic characteristics that predicted academic success among medical students. The non-cognitive characteristics that high achievers possessed as compared to low achievers included positive self-concept and confidence, realistic self-appraisal, leadership qualities, setting long-term goals and academic familiarity [6].
Two useful conceptual frameworks to guide remedial measures come from non-medical contexts. Both these conceptual frameworks explain the possible contributors to student retention in colleges. The framework proposed by Boyles [7] suggests that persistence of students in colleges is influenced either directly or indirectly by background and defining variables, academic self-confidence, academic variables, environmental variables, the degree of social integration, academic outcomes, psychological outcomes, and academic integration [7]. The formula proposed by Seidman [8] summarizes the measures that could be taken to ensure student retention. These include, early identification coupled with early, intensive and continuous intervention.
Medical school attrition was associated with absenteeism in 30%, academic difficulty in 55.7%, social isolation in 20%, and psychological morbidity in 40% of students. Qualitative analysis revealed recurrent themes of isolation, failure, and despair [9]. The evidence from literature suggests that poor academic performance in initial examinations is the best predictor of future academic performance [5]. Therefore, it would be appropriate to target this subset of students for remedial measures. The theoretical underpinnings provided by the conceptual frameworks suggest that it would be beneficial to identify and target specific causes for poor academic performance for remedial measures, rather than non-specific interventions [78]. At present, there is no simple tool to guide this process. The present study proposes to address this gap by using a questionnaire to guide targeted remediation.
The aim of the present study was to assess the effectiveness of a questionnaire to guide targeted remediation among students who failed in the formative examinations in anatomy.
The specific research question that we sought to answer was the following: What are the factors influencing the outcomes of students who fail in the first internal theory examination in anatomy after receiving targeted remediation?
The research objective related to the above question was as follows: to estimate the strength of association between examination results (marks and pass/fail in second theory internal examination) and questionnaire scores (total and domain), first internal examination score, regularity in attending remedial sessions and sex, among students who receive targeted remediation.
Materials and Methods
The study was conducted in the Department of Anatomy at a medical college in South India. Seventy-five students who failed (less than 50% marks) in the first internal theory examination in anatomy were included in the study after obtaining written informed consent. A universal sampling strategy was employed as it would have been unethical to exclude any students from targeted remediation. The age range of the students was from 18 to 25 years. A single group, post-test only study design was used. To guide the targeted remediation process, the students were asked to answer a self-administered questionnaire. The total duration of the study was 5 months (February to June 2017).
Operationalization of variables
The dependent variable was the performance (marks obtained and pass or fail), of the eligible students in the second internal theory examination in anatomy. The independent variables were the total and seven domain scores in the questionnaire obtained both before and after the intervention (targeted remediation), sex of the student, regularity in attending the remedial sessions and the first internal examination score. The regularity of the students in attending the remedial sessions was categorized into regular if they missed no more than one remedial session and irregular if they missed two or more sessions.
Instrument
The questionnaire was designed using the seven steps suggested by Boyd and Westfall (Table 1) [10]. Focus group discussions were conducted with students and faculty to identify domains and facets that were perceived contributors to academic performance. Seven such domains were identified, each explored using five items, making it a 35-item questionnaire (Fig. 1). Each item of the questionnaire had to be answered using a 5-point Likert scale, with a minimum score of 1 and a maximum score of 5. Successive questions were from different domains, such that questions from a domain were equidistant from one another (Fig. 2). The questionnaire was so designed that higher the score in a domain, the greater the problem in that domain. The minimum and maximum possible score for each domain were 5 (1×5) and 25 (5×5), respectively. Domains with a higher score would need to be addressed as a priority. Scores less than or equal to 10 indicated that there was little or no problem in that domain. Domain scores ranging from 11 to 14 indicated some problem, while scores of 15 or more indicated substantial problems. The questionnaire was pilot tested in 26 students. The Cronbach's alpha value of the questionnaire for this pilot test was 0.82.
The questionnaire also had two open ended questions related to examination performance. These questions were “What were the major reasons for your poor performance in the examinations?” and “How do you think these issues could be addressed?” The questionnaire administered after the intervention had different questions. These were, “Has your performance in the anatomy theory examination improved since the 1st internal examination? Describe in as detailed a manner possible as to why you think your performance has improved or not improved.” and “What areas do you think still need to be addressed to further improve your academic performance?”
Intervention
The intervention was done by previously appointed academic mentors from the department of anatomy, each of whom were allotted a group of students. Each mentor had no more than six students under their guidance. The targeted remediation for each domain was standardized by having a two-hour training program for all the mentors. The specific remedial measures that were instituted for each domain are shown in Table 2. Weekly monitored interventions for each student for the specifically identified domains (those with scores of 11 or more) were conducted by the academic mentors for three months till the commencement of the second internal theory assessment examination.
Collection and analysis of the data
Each of the students who failed in the first internal examination were given the questionnaire to answer. The total score as well as the domain specific scores were calculated. The responses to the open-ended questions were also analyzed to try and find specific remedial measures. These specific remedial measures were instituted by the allotted academic mentors. As mentioned previously, the specific remedial measures for each domain were standardized to ensure uniformity of the interventions (Table 2). At the end of the intervention, the regularity of each student in attending the remedial sessions was documented. The students subsequently took the second internal examination and their marks and pass or fail status were collected. They were administered the questionnaire for the second time after the second internal examination and the relevant scores were calculated. The answers to the open-ended questions were collected and thematically analyzed.
The descriptive statistics were the pass rates in the second internal examination and the mean and standard deviations of the marks obtained in the first internal examination, second internal examination, domain specific and total scores (both pre-intervention and post-intervention) of the questionnaire. The proportion of male and female students and those who were regular in attending the remediation sessions was calculated. Group differences between those students who passed in the second internal examination and those who did not, were estimated using the independent sample t test and chi-square test for numerical and categorical variables respectively. A multiple logistic regression was conducted with performance in the second internal examination as the dependent variable and questionnaire scores (pre-intervention and post-intervention), first internal examination marks, sex and regularity of the student as the independent variables. A P-value less than 0.05 was considered statistically significant. As statistical significance was calculated till the third decimal place, any P-values which were less than 0.001 were represented as <0.001.
Results
Of the 75 students who underwent remediation, 54 students passed in the second internal examination. The mean marks in the first and second internal theory examinations of the entire class of 149 students were 46.4±14.4 and 57.1±11.4, respectively. The scores in the second internal examination was found to be significantly higher as compared to the scores in the first internal examination (paired t test, P<0.001). The mean scores in the first internal examination was found to be significantly higher in the non-remedial group (n=74, 58.2±8) as compared to the remedial group (n=75, 34.6±8.3) (independent sample t test, P<0.001). Similarly, the mean scores in the second internal examination was found to be significantly higher in the non-remedial group (62.8±10.6) as compared to the remedial group (51.5±9.2) (independent sample t test, P<0.001). Both the remedial group and the non-remedial group showed a significant increase in marks from the first internal assessment to the second internal assessment (paired t test, P<0.001).
The questionnaire was found to be reliable (Cronbach's alpha=0.86). The total questionnaire score (all 35 items and 7 domains) was found to be significantly lower after remediation (80.1±15 vs. 84.5±14.9, paired t test, P<0.001). The subject related (domain 1) score and study skills (domain 4) score were also significantly lower after remediation (paired t test, P<0.001) (Fig. 3).
The pass rates in the second internal examination among females was 77% (36/47) as compared to 64% (18/28) in males with an odds ratio of 0.55. This difference was not significant (chi-square test, P=0.25). Students who were irregular had a pass rate of 61% (22/36) as compared to those who were regular who had a pass rate of 82% (32/39). The odds ratio was 2.9 and this difference was significant (chi-square test, P=0.048).
Those students who passed the second internal examination had significantly higher marks in both the first internal (37.4±6.7 vs. 27.6±9) and second internal examination (55.9±5.9 vs. 40.3±6) (independent sample t test, P<0.001). Figs. 4 and 5 show a comparison of the questionnaire scores between the students who failed the second internal examination after remediation and those who passed after remediation. The post-remediation study skills (domain 4) score was lower in the group that passed, with a P-value approaching significance (P=0.06). None of the other domains showed significant differences either before or after remediation.
The changes in the internal marks and questionnaire scores among those students who failed in the second internal examination is shown in Fig. 6. The second internal examination marks (40.3±6) were significantly higher than the marks in the first internal examination (27.6±9) (paired t test, P<0.001). The total questionnaire score was less after the intervention (82±16.5 vs. 86.7±17.6) and approached significant levels (paired t test, P=0.07). Subject related difficulties (domain 1) were reduced significantly (paired t test, P=0.007).
The changes in the internal marks and questionnaire scores among those students who passed in the second internal examination is shown in Fig. 7. The second internal examination marks (55.9±5.9) were significantly higher than the marks in the 1st internal examination (37.4±6.7) (paired t test, P<0.001). The total questionnaire score was significantly lower after the intervention (79.3±14.4 vs. 83.7±13.8) (paired t test, P<0.001). Subject related difficulties (domain 1) and difficulties with study skills (domain 6) were reduced significantly (paired t test, P=0.001 and P<0.001, respectively).
A multiple regression was run to predict second internal marks from sex, regularity, first internal marks and pre- and post-total and domain scores on the questionnaire. These variables statistically significantly predicted second internal marks, F(17, 57)=2.5, P=0.006, R2=0.42. Only the first internal marks added statistically significantly (P<0.05) to the prediction. The contribution of the post intervention study skills (domain 6) score to the model approached significance (P=0.07).
A binomial logistic regression was performed to ascertain the effects of first internal marks, sex, regularity in participating in remedial sessions, total questionnaire scores (pre- and post-intervention), individual domain scores (pre- and post-intervention) on the likelihood that the students selected for remediation would pass in the second internal examination. The logistic regression model was statistically significant, χ2(17)=36.9, P=0.003. The model explained 56% (Nagelkerke R2) of the variance in passing the second internals and correctly classified 85% of cases. Higher first internal marks were associated with an increased likelihood of passing in the second internal examination. None of the other independent variables showed any significant associations with passing the second internal examination.
An analysis of the responses to the open-ended questions showed that poor study skills, irregular study patterns, inefficient time management and suboptimal examination skills were perceived to be the reasons for failure in the first internal examination. Potential areas for improvement cited by the students included creation of a study time-table, regular study, allocation of time for revision, participating in academic peer discussions, note taking, taking practice tests, paying more attention in class and stress management. Of the 75 students, 67 (89%) felt that their performance had improved in the second internal examination. The most important reasons for improvement as mentioned by the students after remediation were better study skills, regularity of studying and help from the academic mentor. The responses of the eight students (11%) who felt that they had not improved after the remediation revealed that they were not able to remember what they had studied, did not set aside enough time for studying and revision, were too nervous before examinations to perform well, could not complete the paper and were affected by unforeseen exigencies like illness.
Discussion
The aim of this study was to identify the factors influencing the effectiveness of a targeted remediation strategy for 1st year medical students who underperformed in the 1st internal examination in anatomy. An approach of the early identification of struggling students and institution of specific remedial measures is backed by the existing literature [478111213]. The deleterious effects of underperformance and attrition on the psyche of the student has been well documented [914]. Student failure and attrition places an increased burden on the medical college faculty who are already overburdened with teaching, clinical and administrative responsibilities. Additional financial resources also need to be mobilized to cater to the needs of the underperforming students. It is thus imperative that students who are likely to struggle are identified early in the course so that appropriate corrective action can be taken.
The causes of academic underperformance are multifactorial. These include personal problems, illnesses especially mental, lack of metacognitive skills, an unsupportive environment and reluctance to ask for help [15]. In India, the causes are even more protean due to the heterogenous population groups, number of languages, variations in the syllabi of different pre-medical courses, financial and class disparities and non-uniform admission criteria. College admissions in the field of health professions education are poorly regulated. Many students get admitted into colleges only because their parents are able to pay a capitation fee [16]. While it would be unfair to generalize, students who do gain admission by paying capitation fees are unlikely to come in with a mind-set that is required for a health care professional [17]. Additionally, a sizeable number of students join health professions courses because of parental pressure. The health care field requires professionals who are passionate about their work and not those who are pressured into it. There is also a great variability in the ability of students to cope with the rigors of health professions training, as schools do not adequately prepare students for this. It is in this context that this study has been conducted.
The admissions into medical colleges has been streamlined to an extent by the use of the national eligibility entrance test scores as a sole criterion. The disadvantage is that this is a one-dimensional method of selecting students. The aptitude and the attitudinal prerequisites of a medical student are not considered in the admissions process. As there is no control on the admissions criteria, the first indications of academic difficulties become identifiable only once they join medical college. Of the many predictors of future academic performance in summative assessments, one of the most important is performance in formative assessments [18]. Of these, poor performance in the first semester examination was found to accurately predict future academic difficulties [5]. This was the basis of selecting students based on their first internal examination performance in the current study.
Many strategies like semi-structured interviews, personal learning plans, and reflections have been employed to identify specific areas of deficit that contribute to academic underperformance [192021]. These methods to identify strugglers are difficult to implement in the Indian setting due to the large number of students, their inability to develop personal learning plans and write reflections. Scores on questionnaires related to self-regulated learning, approach to learning and studying and study strategies have shown to be associated with academic performance [222324]. The GHQ-12 was not found to be a useful screening tool to identify future strugglers [5]. The present study utilized a specifically designed questionnaire to guide targeted remediation. It was developed along suggested guidelines and pilot tested to ensure reliability (Table 1) [10]. The validity of the questionnaire was improved since it was developed in the context where the remediation was to take place. The seven domains that the questionnaire explored included benefit from teaching-learning methods, adjusting to a new environment, maintenance of motivation levels, appropriate study skills, ability to manage time effectively, optimum performance in examinations and comfort using English to communicate (Fig. 1).
Many remediation strategies have been used in the past including repeat lectures, small group sessions facilitated by faculty members, the development of personal learning plans, short-integrated study skills programmes, counselling sessions, learner driven remediation strategies and academic mentoring [9202125262728293031]. In the present study, it was decided to use a mentor driven approach to remediation. This approach was selected so that individual attention could be given to each student. To standardize the intervention, a training program was conducted for the mentors to address issues in each of the domains in which problems were identified (Table 2). The perceived benefits to the students from such a mentor driven approach is borne out by their responses to the open-ended questions in the questionnaire, where specific mention was made of the role of the mentors in their improvement.
Of the 75 students who received remediation, 54 (72%) passed in the second internal examination. There is considerable variation in the success of remediation varying from as low as 30% to as high as 91% [19212530323334]. This is due to the heterogeneity in the type and number of students, the remediation strategies and the outcome measures. A majority of previous studies focused on passing an examination after the remediation process. The most common group of medical students in whom remediation was attempted were fourth year students. A number of studies also targeted students in the first two years of medical college [3]. The present study used a before and after study design which is the most commonly used study design to investigate the outcomes of remediation [3].
The class average of the second internal marks was significantly higher than the first internal marks. There may have been many reasons for this improvement including the difficulty of the question paper, level of preparedness of the students, complexity of the portions for the internal examination and remediation offered to the underperforming students. When the marks of the remedial group and the non-remedial group were separately analyzed, it was noticed that the mean scores in non-remedial group were significantly higher both in the first as well as the second internal examination. However, what was striking was the reduction in the gap between the two groups. While there was a 24-mark difference in the mean scores in the first internal examination, this gap had been reduced to around 11 marks in the second internal examination. A part of this improvement in the remedial group could be attributed to the targeted remediation that this group received. Within each group i.e., the remedial and non-remedial group, there was a significant improvement in performance in the second internal examination. The mean scores in the remedial group improved by 17 marks, while the improvement was around five marks in the non-remedial group. This again points to a greater improvement in the remedial group.
The reliability of the questionnaire was shown to be slightly better than when it was pilot tested with fewer students. Before remediation, the domains in which the students had the most difficulties in descending order were study skills, examination skills, time management and subject related issues. This was corroborated by the perceptions of the students collected through the open ended-questions in the questionnaire, thus adding credibility to the results. Many of the students perceived problems in more than one domain. Similar findings were also noted in another study [35]. Previous studies have also shown that underperformers lack the necessary study skills to perform well in examinations. These skills include those related to self-regulated learning, inculcating a deep learning approach, time management and intrinsic motivation levels [222324]. A large majority of the students did not report any problems with motivation, the educational environment and language. There were however a few students who mentioned problems with oral and written communication in English.
After remediation, it was noted that the mean total score, subject related score and study skill score were significantly reduced. This indicated that the students perceived an improvement in the domains mentioned above. Time management and language scores did not change. Scores in the other domains of educational environment, motivation and examination skills were lower, but not significantly so. This suggests that students were maximally benefitted by targeted remediation in study skills and subject related issues (Table 2). The responses of the students to the open-ended questions also suggests that their study skills improved. Many other studies have reported benefits to underperforming students from interventions that target study skills [27323336].
An interesting finding was that there was no significant improvement in the time management and examination skills. This can be explained by the large number of extra-curricular activities that students participate in. These activities compete for time with curricular demands and students find it difficult to strike a balance between the two. The examination skills of the students were dependent on whether the mentor conducted tests or not. It is likely that students who answered these tests could improve these skills as compared to those who did not. This suggests that regular tests must be incorporated into the targeted remediation intervention for examination skills. This has also been emphasized in previous studies [2327].
When a subgroup analysis was conducted comparing those in the remediation group who passed subsequently and those who did not, it was noted that the mean scores in both the first and second internal examination were significantly higher in the former. This difference increased from the first to the second internal examination. This implies that the students who got lower marks in the first internal examination were likely to do so in the second internal examination as well. This finding is similar to other studies where it was found that performance in formative examinations is a good predictor of future academic performance [518].
The difference in the mean questionnaire domain scores in the aforementioned groups was not significant (Figs. 4, 5). However, the difference in the study skills score approached significance, being lower in the group that passed in the second internal examination. This suggests that the latter group were able to derive more benefit from the targeted remediation in this domain. A plausible reason for not observing any significant differences between the groups is the small sample size, especially in the group that failed in the second internal examination.
Within each subgroup, when the results were compared before and after the intervention, it was noted that there was a significant improvement in both groups from the first to the second internal examination. However, in the group that failed in the second internal examination, only the subject related domain score reduced significantly. The reduction in the total questionnaire score also approached significance. In contrast, the other group showed significant reductions in the total questionnaire score, study skills score, examination skills score, in addition to the subject related domain score. This indicates that the more successful students showed significant improvements in more domains as compared to the less successful students.
In this study, the pass rates among females were higher as compared to males. However, this difference was not significant. Once possible reason for the higher pass rates among females was their greater regularity in attending the remedial sessions. Sex differences have also been noted in previous studies, with males being more at risk for underperforming [134]. In the present study, it was noted that students who were more regular in attending the remedial sessions performed significantly better in the second internal examination. This finding is supported by another study in the Netherlands where students who attended more remedial study skill sessions performed better subsequently [33].
The multiple regression analysis showed that the most important predictor of second internal examination marks were the 1st internal examination marks. None of the other predictors showed significant associations, though the association with the post-intervention study skills score approached significance. This suggests that improvement in study skills contributes to better performance in subsequent examinations. The binomial logistic regression analysis corroborated the findings of the multiple regression analysis with only the 1st internal examination marks being significantly associated with passing in the second internal examination. These results point to the multifactorial causes for underperformance among students including academic, social, personal and mental health problems. Even well-planned remediation programs are unlikely to address the non-academic issues to any great extent [3738]
This study has some limitations. An ideal study design to test the hypothesis in the present study would have been a randomized control trial. However, this would raise ethical issues of only one group receiving targeted remediation. A formal sample size estimation was not done in the present study. Again, the ethical issue of some students not receiving the remediation would need to be considered. Even though every effort was taken to ensure standardization of the intervention, there might have been some differences in the way each mentor addressed the problems of the students. This was a possible confounding factor in this study. Even though blueprinting and content validation was done for both the theory papers for the first and second internal examination, there might have been a difference in the difficulty levels of both the papers. Standard setting for both the papers might have overcome this issue. However, standard setting is not routinely done in the Indian context. A possible ethical consideration was the selective identification of a group of students who failed in the first internal examination. This could have potentially stigmatized these students. This was addressed by clearly explaining to the students the reasons for doing this study prior to obtaining informed consent. Another ethical consideration was that only a group of students received the intervention. To circumvent this, a general session on the optimum use of study resources, study skills, time management and answering examinations was conducted for the entire class.
In conclusion, this study provides evidence to show that struggling students perceive a benefit from targeted remediation. The gap between the non-remedial group and the remedial group reduced considerably from the first to the second internal examination. Students who performed better after targeted remediation perceived significant overall improvements, especially in study skills, examination related skills and benefit from teaching-learning methods. However, the students who failed even after remediation showed a significant improvement only in the domain of benefit from teachinglearning methods. These findings suggest the more successful students benefitted from targeted remediation in more domains as compared to the less successful students. The multivariate analysis showed that the first internal marks were the most important predictor of performance in the second internal examination. The improvement in study skills also seems to play an important role in better performance. The analysis of the response to the open-ended questions corroborates the above findings. The findings from this study suggest that an even earlier intervention would prove beneficial to the students. Classes on study skills, time management, examination skills and deriving benefit from teaching-learning methods could be started early and monitored by faculty mentors.
XML Download