PDF
Citation
Print
Introduction
Simulation-based learning (SBL) has been proven to be a cost-effective, easily accessible, and promising educational method in modern education [1]. Numerous studies have proposed it as an effective and safe educational tool for many educational goals, including practice in emergency medicine [1,2]. SBL is a safe environment to practice skills, as it allows participants to repeatedly perform clinical tasks and processes and to debrief about their performance and possible mistakes, without any risk of harming or disturbing a patient. Large meta-analyses and reviews have demonstrated the positive impact of SBL on educational goals and patient-related outcomes [3].
However, given the traditionally oriented “see one, do one, teach one” medical education model in Greece, little is known about the potential impact of SBL on the educational environment and students’ clinical improvement. With that consideration in mind, the Scientific Society of Hellenic Medical Students (SSHMS) organized the first Applied Basic Clinical Seminar with Scenarios for Students (ABCS3), which was a seminar that aimed to improve students’ clinical knowledge and skills through simulation, a strategy that is still rarely implemented in the medical educational system in Greece. In this seminar, specific simulation techniques (partial task simulators, standardized patients, and high-fidelity mannequins) were used in order to demonstrate the environment of certain emergency cases and to force participants to apply their skills in stressful situations. A further goal of the seminar was to familiarize the participants with the techniques and advantages of SBL in undergraduate education, with the ultimate hope that this educational tool will become an integrated part of the undergraduate curriculum in Greece. Although the ABCS3 was approved by the senior academic faculty, it was nonetheless grounded in medical students’ need for a more contemporary and reformed medical educational structure [4]. In fact, medical students suggested most of the core parts of its curriculum, making it a mainly student-led effort. The medical students created a diagram of the scientific modules that would be the core parts of the seminar, and specifically outlined the topics of the ten scenarios and the lectures. They also defined the learning objectives and the evaluation tools of the seminar. The academic professors revised the design process and made appropriate adjustments. Therein lies the critical importance of this initiative; it represented a significant innovation and an attempt for self-improvement on the part of the students, despite the current educational environment of a country in the middle of an economic crisis. The purpose of this study was to describe the novel structure of this seminar and to assess the students’ views of this SBL course as well as any improvement in their clinical knowledge.
Methods
The basic idea of the seminar
The ABCS3 was designed in order to create a simulation-based educational environment that involved the medical students with a variety of common clinical scenarios and familiarized them with fundamental skills and clinical competencies. The seminar lasted for two days and it consisted of three core components: the case-based lectures (Le), the first-aid and resuscitation course (named “ABCDE hands-on station”), and the simulation-based clinical Scenarios (Sc).
A total of 230 medical students from medical schools all over Greece attended the seminar from April 3 to April 4, 2015. Interested students applied online and a scientific committee selected participants based on information from their curricula vitae (year of study, grades in medical school, previously attended courses, and any publications, presentations, or conference abstracts). The attendees who were selected were then split into two groups: the full category, consisting of 60 students (in their fourth, fifth, and sixth years of study) who attended all the modules of the course, and the observer (OB) category, consisting of 170 medical students of any year of study. The OB group included delegates who attended Le, as in the full group, although they observed the Sc in a large lecture hall, via video live-streaming, without having the chance to interact.
The Le core consisted of 15 case-based lectures and was attended before the Sc section, as an introduction that would provide relevant theoretical knowledge and introduce participants to the actual procedures involved in SBL.
The ABCDE hands-on station was held between the Le and the Sc sections, aiming to teach and/or provide a refresher regarding the ABCDE approach to the attendees, as they would be asked to apply it in the subsequent scenarios.
The Sc core was the main part of the seminar and consisted of moulage scenarios and skill stations. Each scenario lasted for thirty minutes, and the 60 participants (from the full category) were divided into 10 groups (I-X) of six people. Each group participated in one scenario (I in Sc1, II in Sc2, etc.). After 30 minutes, all 10 scenarios were completed, and then, with the sound of an electronic alarm, all 10 groups moved forward to the next scenario respectively (I to Sc2, II to Sc3, and so on). Consequently, after 10 cycles of rotation (and 10 repeats of each scenario by different groups), all groups (I to X) had attended all the scenarios (Sc1 to Sc10) (Table 1). Each scenario took place in an appropriately designed room, with all the necessary medical equipment. Mannequins were used in Sc3, Sc5, and Sc9, while in Sc4, Sc6, Sc7, Sc8, and Sc10 trained actors (standardized patients) who were dressed appropriately presented their medical histories and simulated the appropriate symptoms and reactions, according to the specific requirements of each scenario. In the Sc1 station, the delegates practiced suturing, while in Sc2 the participants practiced peripheral and central venous catheter insertion. A laptop was available to visualize lab and imaging test results.
In each repetition, the six participants and two instructors were present in the room. The role of the instructors was to direct the process of the scenarios; to enforce time limits; to help them function as a team, with distinct roles for each team member; and to follow the most appropriate diagnostic and therapeutic strategy to save the patient, within narrow time limits. Table 1 contains details about the topics of the Le and Sc components.
Evaluation
Numerous methods and instruments for assessing undergraduate medical learning environments have been published [5,6]. One of the most widely-used learning environment assessment tools is the Dundee Ready Educational Environment Measure (DREEM). The DREEM and two other instruments (the internal evaluation inventory [InEv] and the MCQs), were used to assess the students’ opinion of the course (DREEM, InEv) and the contribution of the course to their clinical knowledge (MCQs) (Fig. 1). The data were processed using SPSS version 20.0 (IBM Corp., Armonk, NY, USA).
The DREEM inventory
DREEM has been translated into and validated in many languages and has been used for many evaluation purposes such as diagnosing deficiencies in a current educational environment, comparing different groups’ experiences with a given educational environment, and comparing actual experiences of an educational environment with an ideal or expectations within a single group [7,8,9,10,11].
The DREEM inventory consists of 50 five-point Likert questions, with a scale ranging from 0 to 4: 4, strongly agree; 3, agree; 2, uncertain; 1, disagree; and 0, strongly disagree. Questions 4, 8, 9, 17, 25, 35, 39, 48, and 50 are negative statements with inverse scales. The 50-item DREEM has a maximum score of 200, which indicates the ideal educational environment, and a minimum score of 0, which would indicate a very worrying result. The interpretation of the overall score is: 0-50, very poor; 51-100, plenty of problems; 101-150, more positive than negative; and 151-200,excellent. The DREEM has also five subscales, each consisting of a set of question items and with its own interpretation guide: (a) students’ perceptions of learning (SPL), with 12 items and a maximum score of 48; (b) students’ perceptions of teachers, with 11 items and a maximum score of 44; (c) students’ academic self-perceptions (SASP), with eight items and a maximum score of 32; (d) students’ perceptions of atmosphere (SPA), with 12 items and a maximum score of 48; and (e) students’ social self-perceptions (SSSP), with seven items and a maximum score of 28.
All 230 participants were asked to fill in the validated DREEM questionnaire online, with two extra questions (Q1: “Do you think that the course will prove to be beneficial to your clinical skills?” and Q2: “Would you suggest the course to another student?”) following the completion of the course. The reliability of the inventory was tested using Cronbach’s alpha. A descriptive analysis of the data (DREEM, Q1, and Q2) was performed. Moreover, the Mann-Whitney U test was performed to detect if the overall DREEM score and subscale DREEM scores were related to differences in gender, school (Aristotle University of Thessaloniki vs. other universities), and category of participation (full vs. OB). The Kruskal-Wallis test was also used to test any possible difference in DREEM scores according to year of study. Finally, correlations between the Q1, Q2, and DREEM scores were assessed using Kendall’s tau test.
Internal evaluation (InEv) inventory
The InEv inventory was designed as an evaluation tool by the SSHMS and applied specifically to the scenarios of the ABCS3. It consisted of four subsets of 10 questions each and one general question, involving the self-assessment of the respondent’s improvement in clinical skills (ICl). The four subsets referred to four different parameters of the 10 scenarios: adequacy of trainers (ATr), adequacy of facilities and equipment (AFa), adequacy of time (ATi), and general satisfaction (Sat) of each scenario. When applied to all 10 scenarios, this resulted in 40 questions (ATr1, ATr2,..., ATr10, AFa1, etc.) The answers were on a five-point Likert scale, from 1 (minimum) to 5 (maximum). Five more scores were obtained: the trainers’ score (TrS, mean of ATr1 through ATr10, maximum score of 5), the facilities/equipment score (FaS, mean of AFa1 through AFa10, maximum score of 5); the time score (TiS, mean of ATi1 through ATi10, maximum score of 5); the general satisfaction score (SatS, mean of Sat1 through Sat10, maximum score of 5), and the overall score (OvS,=(TrS+FaS+TiS+SatS+4×ICl)/8, maximum score of 5) (Table 2). The TrS, TiS, FaS, and SatS represent the overall score for each of the above four parameters, while the OvS represents the overall score of the seminar, with a special emphasis on ICl, as the improvement of the participants’ clinical skills was the primary goal of the seminar. The 60 full participants were asked to fill out the questionnaire immediately after the completion of the seminar.
The reliability of the InEv inventory was tested using Cronbach’s alpha test. A descriptive analysis of the data and the results was performed. Furthermore, the independent-samples t-test, analysis of variance, the Mann-Whitney U test, and the Kruskal-Wallis test were used to determine whether gender, year of study, or school affected the ICl, TrS, FaS, TiS, SatS and OvS results. Finally, differences in the distributions of Sat1 to Sat10 were tested with the Kruskal-Wallis test, in order to identify which scenarios “generally satisfied” the participants to a significantly greater extent.
MCQs
Tests with MCQs have been broadly used as assessment tools. A norm-referenced test was designed by the scientific committee in order to assess the participants. The 60 students in the full category were asked to answer a set of 22 complex case-based questions in one hour and 20 minutes. For each question, one answer was correct and three were wrong. The same test was distributed twice, once prior to the seminar (pre-test), and again after its completion (post-test).
The descriptive analysis evaluated the characteristics of the examinees, as well as the results of the pre- and post-tests. The related-samples Wilcoxon signed rank test was used to detect improvement or deterioration in the performance between the pre-test and the post-test, and the Mann-Whitney U test and Kruskal-Wallis test were used to examine whether gender, year of study, or school significantly affected performance in the pre- and post-test.
In order to ensure the privacy and confidentiality of the delegates’ personal details, we obtained signed, written consent from them prior to the seminar, which stated that the individuals agreed to fill out these three questionnaires anonymously, without revealing any private information.
Results
DREEM
The response rate was 176/230 (76.5%), with 53 responses (30.1%) from the full group and 123 responses (69.9%) from participants in the OB category. Cronbach’s alpha was ≥0.8 for the overall DREEM score and all subscales (except SSSP), indicating acceptable reliability of the inventory. The descriptive analysis presents the results for each question, as well as for the subscales and the overall score. The overall DREEM score was 144.61 (±28.05), which may be interpreted as more positive than negative. All subscale scores were in the upper range of the second highest level in each category. The mean scores of the extra questions were 3.66 (±0.91) for Q1 and 3.59 (±0.63) for Q2 (Table 3). Gender did not affect the DREEM scores to a significant extent (overall and subscales, P>0.05). School only influenced the SSSP score to a significant extent. However, the category of participation and year of study significantly affected almost all scores, with full participants scoring higher than OB participants on the overall DREEM score (P=0.038), in the SPL subscale (P=0.041), in the SASP subscale (P=0.033), and on Q1, Q2 (P=0.026 and P=0.022, respectively). Among different years of study, fourth-year students scored significantly higher on the DREEM scale than sixth-year students, which was the only significant difference found in overall scores among different years of study, while students in their first, fourth, fifth, and sixth years of study were more willing to suggest the seminar to someone else (Q2). The year of study did not affect Q1 or SPL (Table 4). Finally, Q1 and Q2 responses correlated with the overall DREEM score (Kendall’s tau test) with correlation coefficients of rτ=0.401 (P<0.001) and rτ=0.287 (P<0.001), respectively.
InEv
A total of 56 out of 60 participants (31 males and 25 females) in the full category answered the questionnaire. The reliability of the InEv inventory was tested with Cronbach’s alpha test and found to be acceptable (>0.8 for the entire inventory and the TrS, FaS, and TiS subsets, but not for SatS[0.781]). With 5 as the maximum score, the results were: TrS=4.24 (±0.55), FaS=4.21 (±0.70), TiS=3.94 (±0.82), SatS=4.08 (±0.53), ICl=4.16 (±0.73) and OvS=4.12 (±0.56) (Table 2). No statistically significant differences were found according to gender, years of study, and school (P>0.05). The independent-samples Kruskal-Wallis test revealed statistically significant differences (P<0.001) in general satisfaction among Sc1 through Sc10: Sc2 scored significantly lower than all the other scenarios, except Sc3 and Sc4 (not significantly lower). Sc10 had the highest score, and differed significantly from all the other scenarios, except Sc1, Sc5, and Sc9. Finally, Sc5 had the second highest score, scoring significantly higher than Sc2, Sc3, and Sc5 (Fig. 2).
MCQs
A total of 60 full participants took the MCQs. The pre-test score was 70.08 (±16.23), while the post-test score was 85.53 (±13.40). The independent-samples Wilcoxon signed rank test showed that significant improvement took place (P<0.001). This improvement in performance was also confirmed among subsamples according to gender, year of study, and school (P<0.05 for all tests comparing the pre-test and post-test results). Furthermore, gender and year of study did not affect performance on either the pre-test or post-test, while the students of Aristotle University of Thessaloniki scored significantly higher than those of other universities on the pre-test (P=0.049) and post-test (P=0.016) (Table 5).
Discussion
The ABCS3 seminar included the use of mannequin-based simulators, partial task trainers, and standardized patients. Throughout the entire process, the participants had the opportunity to experience the pressure of the emergency room and to try to develop a diagnostic and therapeutic strategy under these challenging conditions. In this seminar, students improved their clinical knowledge and their ability to take action by diagnosing and intervening. As demonstrated by the pre- and post- MCQs tests, students performed better after the completion of the seminar. This improvement was present among different subsamples.
The participants were attracted to this form of learning and concluded that the course would be substantially beneficial to their clinical skills in the future. The DREEM and InEv questionnaires presented an analytic and comprehensive perspective regarding the strong and weak points of the seminar. The DREEM score was 144.61/200, which may be interpreted as more positive than negative. Students also ranked the first ABCS3 at the upper limit of the second category of each one of the five subscales. First, fourth, fifth, and sixth-year students would recommend the seminar to their colleagues. This may imply that, on the one hand, junior students are attracted by this clinical environment that differed from the strictly theoretical framework of the preclinical first year, while, in contrast, the senior students recognized the value of the seminar, as it offered a different educational experience with the novel aspect of simulation and the benefits thereof, which are not present in the school curriculum. The recognition of the simulation element becomes more obvious when the difference of the DREEM score between the full and OB categories is considered. If we consider the OB category as the control group (having participated in Le and watched the livestreamed Scsession, but not having participated interactively in the hands-on Sc or the ABCDE hands-on station), it scored significantly lower than participants in the full category in most of the DREEM, Q1, and Q2 scores (including the overall DREEM score), suggesting that the simulation-based hands-on aspect of the seminar made a significant difference. The overall score of the InEv inventory (4.12±0.56/5), as well as the subscores (TrS= 4.24, FaS=4.21, TiS=3.94, SatS=4.08, and ICl=4.16) also indicated the high quality of the seminar. A comparison of the general satisfaction score among the scenarios (SatSc1-10), showed that Sc10 (orthopedics scenario) and Sc5 (cardiology scenario) scored especially well, whereas it is necessity to update and improve Sc2 (peripheral and central venous catheterization station), which recorded the lowest score.
Taking into account that evaluations of this type rarely result in the alteration of educational structures, the primary goal of the evaluation procedures of this first attempt was to use the feedback to improve the structure of the second ABCS3 [12,13]. Moreover, we will improve the evaluation tools and procedures used for the second ABCS3 and then compare the new results to those of the first ABCS3. Last but not least, one further goal is to contribute to the integration of simulation techniques in the medical school educational curriculum, in order to improve its efficacy and the students’ perception thereof.
XML Download