PDF
ePub
Citation
Print
INTRODUCTION
Infectious disease outbreaks such as malaria and human immunodeficiency virus (HIV)/acquired immune deficiency syndrome (AIDS) have resulted in growing demand for medical care in sub-Saharan Africa. In recent years, incidence of road traffic injuries and cardiovascular disease has also increased due to rapid urbanization, industrial development, and health behavior changes. However, emergency medical services (EMS) system is still underdeveloped, and mortality and disability from acute medical conditions and injuries are markedly higher compared to developed countries (12). The rising burden of acute medical conditions puts more emphasis on emergency medical care in prehospital as well as hospital settings (3).
In 2009, the African Federation for Emergency Medicine (AFEM) was established with joint efforts of South Africa, Ghana, and Ethiopia, to develop emergency care system in partnership with advanced countries. In addition, countries including Madagascar, Rwanda, Sudan, Uganda, and Zambia developed residency training programs in emergency medicine (4). Through international partnerships, several African countries tried to strengthen emergency care system by training local directors in emergency medicine, developing training modules, establishing an emergency training center, and creating an academic training program for residents and nurses (4567).
Development of emergency care system in Cameroon, however, is still in a nascent stage. Total health expenditure in Cameroon was a 5% of the gross domestic product during the past 10 years, and poor coordination between stakeholders impeded quality development of services in emergency departments (8). Recently, the Cameroonian government has taken an initiative to establish training schools for emergency medicine to address lack of health resources, revise education curricula, promote permanent employment of healthcare workers in the public sector, and increase funds from external sponsorships (9).
Well-trained emergency physicians serve as key human resources in emergency care system. Emergency physicians should have well-rounded competence in both the diagnosis and treatment of emergency patients, be able to manage patients, effectively utilize emergency resources, cooperate with local community, and develop and collaborate with prehospital EMS system. In Cameroon, however, residency training program in emergency medicine has not yet been implemented, and development of a training curriculum to strengthen the competence of emergency physicians remains as an urgent task for providing reliable emergency care.
Competence is the ability of an individual to apply one's knowledge, understanding, skills, and judgment in performing effectively in the field of professional practice (10). Development of a competence-based curriculum has become central to the education and training of healthcare professionals (1112). However, tools to evaluate competence of emergency personnel have not yet been developed. Therefore, this study aimed to develop a competence assessment tool in emergency medicine and to validate it by applying the tools to Cameroonian healthcare professionals.
MATERIALS AND METHODS
Study design and setting
This was a cross-sectional, descriptive study. The Korea International Cooperation Agency (KOICA), which is a government-run bilateral aid agency in Korea, has progressed a project with the Cameroon Ministry of Public Health to construct the Yaoundé National Emergency Center. Between October 27 and 31, 2014, a short-term educational program was provided to 58 healthcare professionals (13 specialists, 10 general physicians, and 35 nurses) in the Central Hospital of Yaoundé who are expected to work at the Yaoundé National Emergency Center after its opening in 2015. The educational program for doctors included Advanced Trauma Life Support (ATLS), emergency ultrasound, electrocardiogram course, etc.
In Cameroon, medical student educational program is a 7-year course including a 1-year research activity at a college; and continuous medical education programs are a 2-year general physician training and a 4-year residency programs. For emergency medicine, a 2-year special training program is offered but is not recognized as a certified residency program.
Development of competence evaluation tools
We developed a self-survey questionnaire and a scenario-based assessment tool to evaluate competence in clinical knowledge and self-confidence to perform clinical practices or procedures.
The self-survey questionnaire was developed based on the European curriculum for emergency medicine, core curriculum of the International Federation for Emergency Medicine, learning objectives of the Korean Academy of Medical Sciences, and emergency resident training programs of Korean Society of Emergency Medicine (1314). It is composed of a total of 113 questions: 17 domains with 82 questions about core clinical knowledge, 4 domains with 11 questions about specific aspects of emergency medicine, and 2 domains with 20 questions about clinical procedures and skills.
A two-step pretest was conducted to improve the validity of the competence assessment tool. The first step involved an expert consensus survey with 7 Korean emergency physicians who have previously contributed to the educational program for healthcare professionals in Yaoundé. Questions in which more than 4 emergency specialists reached consensus were selected as appropriate items for the survey, and those in which more than 4 emergency specialists found confusing were modified as appropriate. As a result of the expert consensus survey, a total of 94 questions were selected as potential candidates of the competence assessment tool; seventeen questions on core clinical knowledge and two questions on clinical procedures and skills were excluded. The second step assessed the discrimination power of each question between junior (1st or 2nd year) and senior (3rd or 4th year) emergency residents. Korean junior (n = 7) and senior (n = 5) emergency residents participated in the evaluation. All 94 questions in the 23 domains showed high discrimination power, and as a result, were selected for the competence assessment tool for emergency healthcare professionals in Yaoundé (Table 1). Full results of the two-step pretest are reported in Supplementary Table 1.
Table 1
Comparison of self-survey competency assessment results in Korean emergency residents and Cameroonian healthcare professionals
To test validity of the self-survey and to objectively assess competence in emergency medicine, we also developed a scenario-based competence assessment tool based on a simulation workbook and a relevant web site (151617). Five life-threatening diagnoses frequently presenting in the emergency room of the Central Hospital of Yaoundé were selected as scenario topics: ST elevation myocardial infarction (STEMI); diabetes ketoacidosis (DKA); septic shock due to necrotizing fasciitis; multiple trauma by road traffic injury; and anaphylaxis. For these scenarios, 14 questions were chosen among the self-survey questions, and a total of 26 sub-questions (3 for STEMI, 6 for DKA, 11 for septic shock, 3 for multiple trauma, and 3 for anaphylaxis) and answers were also developed to assess the competence in diagnosis and treatment of the 5 diseases. Detailed information of the scenario-based competence assessment tool is reported in Supplementary Table 2.
Study population and study protocols
Eligible population was Cameroonian doctors (13 specialists and 10 general physicians) who had taken the education program in 2014. Competence was assessed in two steps with healthcare professionals who agreed to participate in the study.
The first step of competence assessment was conducted in November 2014 using a 17-page self-survey on clinical knowledge and self-confidence to perform clinical practices or procedures. The survey took roughly 40 minutes to complete including an introduction to purpose and methods of the survey. Each question was answered in a 5-point Likert scale. All results were coded using Microsoft Excel (ver. 14.0, Microsoft®, Los Angeles, CA, USA) by an appointed data entry clerk.
The second scenario-based assessment was conducted with 6 doctors (3 general physicians and 3 specialists) who had previously participated in the self-survey on December 10 and 11, 2014, in the form of an individual interview with two Korean emergency physicians. One emergency physician explained a scenario, and the physicians evaluated the answers in which a Cameroonian respondent provided about his or her medical knowledge on patient assessment, interpretation, diagnosis, and treatment. For increased objectivity, both emergency physicians independently assessed the respondent's answers on a 5-point Likert scale.
Main outcomes
Main outcome was competence in emergency medicine, captured by self-survey and scenario-based competence assessment on a 5-point Likert scale: 1) don't know at all; 2) don't know; 3) average; 4) know; and 5) know well. Secondary outcomes were agreement rate and difference of scores between the two methods of self-survey and scenario-based competence assessment.
Statistical analysis
Descriptive statistics of the self-survey and scenario-based competence assessment results on a 5-point Likert scale were expressed with means and standard deviations (SDs). Differences of assessment results between groups were compared using the Student's t-test.
The power of discrimination for the self-survey between junior and senior residents was measured as the area under the generated receiver operating characteristic (ROC) area under curve (AUC); and AUC greater than 0.7 was said to have high discrimination power. For the scenario-based competence assessment, inter-rater reliability between the two Korean emergency physicians was analyzed using the weighted kappa.
For every doctor participating in both competence assessment methods, we compared the score difference of all 14 questions between the first self-survey and the second scenario-based competence assessment. The difference between the two assessment results was analyzed using paired t-test, agreement rate, and weighted kappa. When the difference between the result of the first self-survey and the second scenario-based assessment was within one point, it was said that the two results had an agreement. Level of statistical significance was defined as P < 0.05.
RESULTS
The first competence assessment: self-survey
Among a total of 23 doctors, 10 specialists and 9 general physicians participated in the first competence assessment (82.6%). Mean (SD) age of the 19 participants was 33.1 (5.6). And 6 (31.6%) were male. Main areas of practice for the 10 specialists included anesthesia (n = 3), internal medicine (n = 3), general surgery (n = 2), gynecology and emergency medicine (n = 1), and laboratory medicine (n = 1).
In the self-survey competence assessment, the Cameroonian healthcare professionals scored the highest in the domain of infectious disease (mean ± SD, 4.6 ± 0.4), followed by obstetrics and gynecology (4.2 ± 0.6) and hematology and oncology (4.2 ± 0.5); in contrast, they scored the lowest grade in the domains of disaster, abuse and assault, and psychiatric and behavior disorder (each with a mean of 2.8). Domains of cardiovascular, musculoskeletal, procedure, resuscitation, cardiopulmonary resuscitation (CPR) skills, and environmental injuries had significantly higher scores in specialists compared to general physicians (all P < 0.05). The average scores in general physicians were higher than specialists only in the 3 domains of obstetrics and gynecology; eyes, ear, nose, throat, oral, and neck; and psychiatric and behavior; however, the score differences were not statistically significant (Table 1).
The second competence assessment: scenario-based
For the 14 questions of the scenario-based competence assessment which were answered by the 6 study participants, the inter-rater reliability between the two interviewers had a weighted kappa 0.88 and 95% confidence interval (CI) 0.82–0.95 (Table 2). Highest scores of the scenario-based assessment were observed in the knowledge of multiple trauma (mean ± SD, 4.3 ± 1.2), followed by anaphylaxis (3.4 ± 1.4), DKA (3.3 ± 1.0), and STEMI (2.5 ± 1.4). Knowledge of septic shock scored the lowest in the scenario-based assessment (mean ± SD, 2.2 ± 1.1) (Table 3).
Table 2
Inter-rater reliability between two interviewers (Korean emergency physicians)
| Interviewer | Interviewer 2 | ||||||
|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | Total | ||
| Interviewer 1 | 1 | 12 | 2 | 0 | 1 | 0 | 15 |
| 2 | 1 | 12 | 3 | 0 | 0 | 16 | |
| 3 | 0 | 2 | 13 | 2 | 0 | 17 | |
| 4 | 0 | 0 | 2 | 18 | 0 | 20 | |
| 5 | 0 | 0 | 0 | 0 | 16 | 16 | |
| Total | 13 | 16 | 18 | 21 | 16 | 84* | |
Table 3
Scenario-based competency assessment between GPs and specialists in Cameroon
Comparison between first and second competence assessment
Mean difference between the self-survey and scenario-based assessment was statistically insignificant (mean, −0.02; 95% CI, −0.41 to 0.36). In terms of individual participants, the mean difference (95% CI) between the scenario-based assessment and the self-survey ranged from −1.21 (−2.15 to −0.28) to 0.64 (−0.25 to 1.54) (Table 4). Results of the 14 questions in the self-survey and the scenario-based assessment showed 58.3% agreement rate, 17.9% under-estimation rate, and 23.8% over-estimation rate (Table 5).
Table 4
Difference between scenario-based test and self-survey test
DISCUSSION
This study involved development of a self-survey and a scenario-based competence assessment tool for emergency medicine using literature review, expert consensus, and a discrimination test with pilot survey. We administered the self-survey to Cameroonian healthcare professionals and later conducted an objective, scenario-based competence assessment to test validity of the self-survey. Mean difference between the self-survey and scenario-based assessment was negligible, and scores between two assessments showed moderate agreement rate (58%). Competence-based education has become an important key of curriculum for healthcare professionals, which highlights the needs for reliable, valid, and feasible competence assessment tools in mid- and long-term education courses (181920). The areas of practice which showed poor competence in this study should comprise the main contents of emergency medicine curriculum for the Cameroonian healthcare professionals.
The self-survey competence assessment is one of the most formative and summative forms of evaluation. Repetitive self-surveys can benefit both learners and educators through continuous monitoring of self-competence and education impacts, respectively. In this study, the Cameroonian participants showed high competence in the domains of infectious disease, obstetrics and gynecology, hematology and oncology. Specialists who had completed advanced education and training showed high competence in the domains of clinical procedure, CPR skills, and resuscitation. However, all Cameroonian participants had relatively low competence in specific aspects of emergency medicine including disasters and prehospital care. There are specific aspects of emergency medicine as well as clinical skills and procedures which function as key competence of an emergency physician (14); therefore, those domains which showed poor competence should be of particular interest for the development of emergency medicine curriculum for the Cameroonian healthcare professionals. Furthermore, periodic monitoring of physicians' competence can also be utilized to evaluate the effects of curriculum in the future.
The self-survey has its advantage of convenience and cost-effectiveness, but its results may be subjective. When the self-survey is combined with objective assessment methods, we may expect to obtain more valid and practical results (1421). In this study, we developed scenario-based assessment tools for the 5 life-threatening diseases which frequently present in the emergency rooms of Cameroon. When comparing results of the subjective self-survey and the objective scenario-based competence assessment, the scores were comparable with mean difference of −0.02 (−0.41 to 0.36) and had a moderate agreement rate of 58.3%. However, one of the study participants overrated his or her own competence with self-survey than did in the scenario-based assessment, and the calculated overestimation rate (23.8%) was higher than the underestimation rate (17.9%). In previous literature, the correlations between self-survey and externally observed measurement of competence had been controversial. Some studies reported that self-assessment for selected fields or categories was a reliable predictor of clinical performance (15222324). In another meta-analysis, however, only 35% of studies showed positive correlation between self-assessments and external assessments (2125). In our study, self-surveys had a moderate correlation with the externally observed measurement. Methodologic quality of a self-survey, knowledge level and training experience of a respondent, and quality and skills of an evaluator are important factors to consider for an accurate evaluation of competence using self-survey.
Self-survey assessment in medical knowledge should accurately measure competence as well as actual performance as reflected by an external assessment (18). There are many external, objective competence assessment methods that compensate for the weaknesses of a self-survey, such as oral examinations, procedure tests, simulation examinations, objective structured clinical examination (OSCE), standard patient examinations, and clinical record reviews (1418). In this study, we developed self-survey and scenario-based oral examination which were designed to assess competence of clinical performance in an environment which resembles an actual situation. Furthermore, we applied both assessment tools to the Cameroonian healthcare providers, and results of the two showed a moderate correlation. Using the two competence assessment tools, we expect to develop curriculums for emergency medicine that focuses on specific aspects of emergency medicine and clinical skills and procedures in which the Cameroonian healthcare professionals had poor competence on.
There are several limitations in this study. First, only 14 selected questions among the 94 self-survey questions were used in the scenario-based test. Although the selected topics were based on their high frequency of incidence in Cameroon, not all questions in the self-survey were evaluated with the objective scenario-based assessment, resulting in limited validity of the self-survey. Second, since Korea and Cameroon have different medical environment, there is a possibility that some questions were not considered as important and were excluded during the process of expert consensus and discrimination test in Korea. Supplementary assessment such as clinical behavior observation might be helpful to improve the validity of competence evaluation tool. Third, there was a one month gap between the administration of self-survey and scenario-based assessment. Therefore, we could not measure if there had been any individual efforts to improve competence in the interim.
In conclusion, we developed and administered self-survey and objective scenario-based competence assessment tools to evaluate the competence in emergency medicine among Cameroonian healthcare professionals. Results of the two evaluation tools showed a moderate correlation, and study participants showed relatively low competence in specific aspects of emergency medicine and clinical procedures and skills. Results of this assessment can be used to develop curriculums for emergency medicine which is tailored to the needs of Cameroonian healthcare professionals. The assessment tools can be used as valuable resources for assessing competence and developing need-based curriculum for emergency medicine in developing countries including those in Africa.
XML Download