Varying levels of difficulty index of skills-test items randomly selected by examinees on the Korean emergency medical technician licensing examination

Bongyeun Koh; Sunggi Hong; Soon-Sim Kim; Jin-Sook Hyun; Milye Baek; Jundong Moon; Hayran Kwon; Gyoungyong Kim; Seonggi Min; Gu-Hyun Kang

doi:10.3352/jeehp.2016.13.5

Abstract

Purpose:

The goal of this study was to characterize the difficulty index of the items in the skills test components of the class I and II Korean emergency medical technician licensing examination (KEMTLE), which requires examinees to select items randomly.

Methods:

The results of 1,309 class I KEMTLE examinations and 1,801 class II KEMTLE examinations in 2013 were subjected to analysis. Items from the basic and advanced skills test sections of the KEMTLE were compared to determine whether some were significantly more difficult than others.

Results:

In the class I KEMTLE, all 4 of the items on the basic skills test showed significant variation in difficulty index (P<0.01), as well as 4 of the 5 items on the advanced skills test (P<0.05). In the class II KEMTLE, 4 of the 5 items on the basic skills test showed significantly different difficulty index (P<0.01), as well as all 3 of the advanced skills test items (P<0.01).

Conclusion:

In the skills test components of the class I and II KEMTLE, the procedure in which examinees randomly select questions should be revised to require examinees to respond to a set of fixed items in order to improve the reliability of the national licensing examination.

Introduction

In Korea, there are 2 kinds of emergency medical technicians: class I and II. Class I emergency medical technicians are comparable to paramedics, while class II includes basic emergency medical technicians. Examinees of the class I Korean emergency medicine technician licensing examination (KEMTLE) should be college or university graduates who have completed a 3- or 4-year emergency medicine technician course. Graduates of the fire service academies, which have a six-month training period, are eligible to take the class II KEMTLE. When a class II emergency medical technician has acquired more than 3 years of experience in firefighting, he or she is eligible to take the class I KEMTLE. The KEMTLE is supervised by the National Health Personnel Licensing Examination Board (NHPLEB) of Korea. Both the class I and II KEMTLE are composed of written and skill-based tests. The skills tests of both examinations are composed of a basic skills test, an advanced skills test, and physical test items. Examinees randomly select 1 item each from the basic and advanced skills test item pools. The skills test for each examinee is scored by a grader according to a checklist. Table 1 presents the composition of the KEMTLE and the topics of the skills tests. Concerns have been raised regarding the reliability of item selection by examinees. Administering the same test items to all examinees is ideal from the point of view of test reliability; however, this approach has not been applied to the KEMTLE due to limited time, human resources, and budget. This study aimed to determine whether the skills test items that examinees randomly select differ in difficulty, by analyzing examination data from the 2013 class I and II KEMTLE examinations provided by the NHPLEB.

Methods

Study design

This study involved the cross-sectional analysis of examination data without any personal identifiers.

Materials and subjects

Skills test data from 1,309 class I and 1,801 class II examinees from the 19th KEMTLE in 2013 were obtained from the NHPLEB. These data were used to compare the difficulty index of each item in the basic and advanced skills tests. The difficulty index of each item was calculated as each examinee’s scores divided by the maximum score; therefore, a high difficulty index means that examinees performed well on the question.

Statistical analysis

The difficulty index of items was compared using SPSS version 18.0 (SPSS Inc., Chicago, IL, USA). One-way analysis of variance and the t-test were used to compare the means. The cutoff for statistical significance was set at P<0.05. Items that exhibited significant intergroup differences were further analyzed using the post-hoc Scheffe test.

Ethical approval

This study was conducted after receiving individual consent from all subjects, each of whom had previously received an explanation of the study’s objectives.

Results

Class I KEMTLE

The average difficulty index of the 4 basic skills test items was 0.87 (Table 2). Item D (the assessment of internal medicine patients) showed the highest difficulty index (0.91), while item A (endotracheal intubation) showed the lowest (0.85). The 4 items of the basic skills test significantly varied in terms of difficulty index (P<0.01). Post-hoc testing indicated that the difficulty index of item D was different from that of all other items: A, B (automated external defibrillator therapy), and C (intravenous injection) (P<0.01). The average difficulty index of 4 advanced skills items was 0.55. Item H (laryngeal tube intubation) showed the highest average score (0.56). Item I (electrocardiogram check and reading) showed the lowest item difficulty index (0.53). The 4 items of the advanced skills test showed significant differences in difficulty index (P<0.05) (Table 2). Post-hoc testing indicated that the difficulty index of item I was different than that of items E (infant airway obstruction therapy), F (primary assessment of trauma), and G (traction splint application) (P<0.01).

Class II KEMTLE

The average difficulty of the 5 items of the basic skills test was 0.77 (Table 3). Item J (automated external defibrillator therapy) was the least difficult (0.85), while item K (primary assessment of trauma) was the most difficult (0.73). The 4 items in the basic skills test showed significantly different levels of difficulty index (P<0.01). Post-hoc testing indicated significant differences between items J and K, J and L (traction splint application), J and M (infant airway obstruction therapy), J and N (laryngeal tube intubation), and K and M in the basic skills test items (P<0.05). The average difficulty index of the 3 advanced skills test items was 0.76. Item G (suction and oxygen therapy) was the least difficult (0.84), while item O (infant cardiopulmonary resuscitation) was the most difficult (0.68). The 3 advanced skills test items significantly varied in difficulty index (P<0.01). Post-hoc testing indicated significant differences among items O, P (suction and oxygen therapy), and Q (vacuum splint application) in the advanced skills test (P<0.01) (Table 3).

Discussion

The KEMTLE permits examinees to randomly choose test items, which they then solve in front of the graders. This study demonstrated that the random selection of items by examinees requires reform, since differences in item difficulty index among items can affect examinees’ total scores on the licensing examination. In our comparison of the difficulty index of skills test items, we made the assumption that the ability of the examinees who selected each item was evenly distributed. If an examinee selects a skills test item with a high item difficulty, he or she can achieve a better score. Ideally, implementing fixed items would solve this problem, as is already done on the national certification examination for emergency medicine technician-paramedics in the United States [2]. Out of the 25 national health personnel licensing examinations supervised by the NHPLEB of the Republic of Korea, the random selection of skills test items by examinees is present in two: the KEMTLE and the Korean dental hygienist licensing examination. If a skills test center is established for the health personnel licensing examination, the present performance evaluation system allowing examinees to randomly select skills test items could easily be reformed. We believe that the Korean government should support the establishment of skills test centers that can be used for skills tests in a variety of health personnel licensing examinations. Fortunately, the city of Daegu announced that a health skills test training center would be constructed in the Daegu Innovation City Cluster by 2020, since the plans of this center passed the preliminary validity evaluation by the Ministry of Strategy and Finance of the Korean government (http://www.daegu.go.kr/Contents/Content.aspx?cid= 109:17 [cited 2016 Jan 1]). We hope that the construction of this center is completed in the near future and that it enters into active use.

Improvement of the content of the KEMTLE is also required, as the current examination technique is simple and does not evaluate the problem-solving and comprehensive thinking skills that emergency medicine technicians must possess in emergency medical situations [3]. Previous research has also argued for the inclusion of integrated management questions in skills tests [1]. Measures for developing sophisticated yet simple administration formats for skills tests should therefore be studied. Recently, the importance of on-site skills test has been increasingly emphasized. Assessment methods using simulations have proven effective for performance-based evaluations [4]. Similarly, simulation- and scenario-based skills test items should be developed and used in national licensing examinations in order to adequately reflect the changing environment of emergency medical technician services.

In conclusion, a fixed set of items should be administered in order to overcome the aforementioned limitations posed by varying levels of difficulty index in the test items that are randomly selected by examinees.

Notes

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

This research was supported by a research grant from the National Health Personnel Licensing Examination Board of Korea (Fundref ID: 10.13039/501100003647) (2014).

Supplementary materials

Audio recording of the abstract.

References

1. Yoou SK, Choi ES, Sin DM, Jo JM, Lee JE, Uhm TH, Roh SG, Kim YS. Proposal of a new emergency medical technician national practical examination. Korean J Emerg Med Serv. 2012; 16:45–62.

2. National Registry of Emergency Medical Technicians. Paramedic national registry skills test [Internet]. Ohio (USA): NREMT;2015. [cited 2015 September 1]. Available from: https://www.nremt.org/nremt/about/reg_para_history.asp.

3. National Health Personnel Licensing Examination Board. EMT practical examination notice [Internet]. Seoul (KR): National Health Personnel Licensing Examination Board;2015. [cited 2015 September 1]. Available from: http://www.kuksiwon.or.kr/index.aspx.

4. Tavares W, LeBlanc VR, Mausz J, Sun V, Eva KW. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehosp Emerg Care. 2013; 18:116–122. http://dx.doi.org/10.3109/10903127.2013.818178.

Table 1.

Subjects, number of items, score allocation, total score, and item types on the Korean emergency medical technician licensing examination (KEMTLE), 2013

Certification type		Subjects	No. of items	Score allocation	Total score	Item type
Class I KEMTLE-Paramedic	Written test	Preclinical medicine	40	1 per item	240	Multiple choice
		Management of an emergency patient	40
		Introduction to advanced paramedic care	40
		Special advanced paramedic care	90
		Emergency medical law	30
	Basic skills test	Intubation	1 item randomly selected out of 4 by examinees	30 per item	60	Assessment of proficiency
		Automated external defibrillator therapy
		Intravenous injection
		Patient assessment
	Advanced skills test	Infant airway obstruction	1 item randomly selected out of 5 by examinees	20 per item
		Primary assessment of trauma
		Traction splint application
		Laryngeal tube insertion
		Electrocardiogram check and reading
	Common physical test	Back muscle strength	1	10 per item
Class II KEMTLE-Basic	Written test	Introduction to basic emergency care	30	1 per item	150	Multiple choice
		Management of an emergency patient	30
		Emergency medical law	20
		Special basic emergency care	50
		Emergency equipment	20
	Basic skills test	Automated external defibrillator therapy	1 item random selected out of 5 by examinees	30 per item	60	Assessment of proficiency
		Primary assessment of trauma
		Traction splint application
		Infant airway obstruction therapy
		Laryngeal tube insertion
	Advanced skills test	Infant cardiopulmonary resuscitation	1 item randomly selected out of 3 by examinees	20 per item
		Suction and oxygen therapy
		Vacuum splint application
	Common physical test	Back muscle strength	1	10 per item

Table 2.

Comparison of the difficulty index of the randomly selected items on the class I Korean emergency medical technician licensing examination in 2013

Section	Items	No. of examinees	Item difficulty index		F	Scheffe Post-hoc
Section	Items	No. of examinees	Mean	Standard deviation	F	Scheffe Post-hoc
Basic skills test	A	306	0.85	0.18	10.688^a)	A = B, A = C,
	B	291	0.86	0.14		A ≠ D, B = C,
	C	341	0.86	0.17		B ≠ D, C ≠ D,
	D	371	0.91	0.16
	Total	1,309	0.87	0.16
Advanced skills test	E	265	0.56	0.08	4.074^b)	E = F, E = G,
	F	257	0.56	0.09		E = H, E ≠ I,
	G	301	0.55	0.10		F = G, F = H,
	H	260	0.56	0.11		F ≠ I, G = H,
	I	226	0.53	0.13		G = I, H ≠ I,
	Total	1,309	0.55	0.10

A, endotracheal intubation; B, automated external defibrillator therapy; C, intravenous injection; D, assessment of internal medicine patients; E, infant airway obstruction therapy; F, primary assessment of trauma; G, traction splint application; H, laryngeal tube intubation; I, electrocardiogram check and reading.

^a) P<0.01

^b) P<0.05

Table 3.

Comparison of the difficulty index of the randomly selected items on the class II Korean emergency medical technician licensing examination in 2013

Section	Items	No. of examinees	Item difficulty index		F	Scheffe Post-hoc
Section	Items	No. of examinees	Mean	Standard deviation	F	Scheffe Post-hoc
Basic skills test	J	366	0.85	0.13	27.884^a)	J ≠ K, J ≠ C,
	K	365	0.73	0.20		J ≠ M, J ≠ N,
	L	344	0.75	0.18		K=L, K ≠ M,
	M	409	0.78	0.15		K=E, L ≠ M,
	N	317	0.75	0.20		L = N, D = N,
	Total	1,801	0.77	0.18
Advanced skills test	O	614	0.68	0.15	116.062^a)	O ≠ P, O ≠ Q,
	P	606	0.84	0.10		P ≠ Q
	Q	581	0.76	0.13
	Total	1,801	0.76	0.13

J, automated external defibrillator therapy; K, primary assessment of trauma; L, traction splint application; M, infant airway obstruction therapy; N, laryngeal tube intubation; O, infant cardiopulmonary resuscitation; P, suction and oxygen therapy; Q, vacuum splint application.

^a) P<0.01