This study used a mixed-methods design combining quantitative and qualitative approaches. First, a perception survey was conducted using a questionnaire; targeting nursing professors and clinical nurses with more than 5 years of work experience, the survey investigated opinions on the desirable aspects of a performance exam and operational plans, such as ways to implement a performance exam, advisable forms of the test, abilities that need to be tested, nursing skills to be evaluated, ways to manage the test, and when the test should be administered.

The second method was to conduct focus group interviews (FGIs). Using a semi-structured questions, the opinions of nursing education experts—college professors, clinical instructors, and on-site nurse leaders at hospitals—were investigated in depth.

Questionnaire survey: The questionnaire survey targeted nursing college professors, clinical nurses, and experts in nursing education policy. Among the institutions of nursing education and nursing policy (200 nursing colleges, 20 general hospitals with more than 300 wards, and KABONE), the survey selected its respondents by convenience sampling based on the regional distribution of nursing colleges and general hospitals. The survey took 20–30 minutes to complete, and participants were rewarded with gifts worth 10 dollars. Roughly 250 copies of the questionnaire were distributed, 202 were collected, and 200 were analyzed, since 2 of them were deemed not to contain sincere responses.

Our research team analyzed the subjects of foreign national examinations (including performance exams), the status of domestic practical training, and the status of the core nursing skills training/education presented and validated by KABONE [4], and the questionnaire about the content and methods of a performance exam was developed on this basis. The items in the questionnaire were as follows: validity of the nursing skills to be included in the test, the test implementation method, the most preferable type of the test, abilities to be evaluated, the operating mode of the test, the timing of test implementation, the number of evaluators, public release of test items, a reasonable test fee, the proper number of questions, and the passing criteria. The validity of the nursing skills to be included in the test was evaluated on a 4 point Likert scale: 1, not valid at all; 2, not valid; 3, valid; 4, very valid, and the Cronbach’s alpha was 0.92.

Focus group interviews: FGIs were conducted in order to collect the opinions of practitioners such as nurses, professors, and clinical instructors. FGIs were done with 28 participants, all of whom were professors with equal to or more than 2 years of teaching experience or clinical nurses with equal to or more than 5 years of work experience from various regions. Every group consisted of professors, clinical instructors, and hospital nurse leaders, who were all interviewed by a researcher. Each interviewee received an explanation of the purpose of the study, the process, and the methods of data collection before the interview, and they were all notified of their right to refuse further participation at any time during the interview. They all submitted a written consent form. Data were collected from March to June 2013 in the form of recorded audio files, transcripts of the recordings, and post-interview filed notes written by researchers. The following semistructured questions were asked in the FGIs:

Using IBM SPSS ver. 19.0 (IBM Corp., Armonk, NY, USA), the survey results were analyzed in terms of frequency, percentage, mean value, and standard deviation. The qualitative data collected through FGIs were categorized and the themes were extracted through content analysis.

In terms of ethical considerations regarding the research participants, the survey and the interview were conducted only when the participant understood the purpose of the research and voluntarily signed a written informed consent form to participate in the research. The written consent form guaranteed the participant’s anonymity, and explained that all data would be used only for research and would be destroyed after the study.

General characteristics of the participants: Of 200 survey participants, 50% were college professors, and 39.1% were clinical nurses. The participants’ average age was 44.34 years, average teaching career 12.42 years, and average clinical career 10.61 years. In terms of geographical distribution, 19.3% of them worked in Seoul, 11.9% in Gyeonggi Province, 14.4% in Chungcheong Province, and 31.2% in Gyeongsang Province (Table 1). Raw data were available from Supplement 1.

Validity of the nursing skills to be included in the clinical performance examination: The nursing skills to be included in the performance exam were evaluated on a 4-point Likert scale, and the item that scored the highest validity was measuring vital signs (mean, 3.53; standard deviation, 0.77), followed by indwelling catheter insertion, endotracheal suction, intramuscular injections, nasal cannula oxygen therapy, catheterization, subcutaneous injections, enteral nutrition via nasoenteric tubes, oxygen saturation measurements and electrocardiogram monitoring, basic cardiopulmonary resuscitation and defibrillator use, and intravenous therapy (Table 2).

Action plans for the clinical performance exam: Of the respondents, 71.3% chose a hands-on performance exam on simulators or standardized patients as the most suitable format for the performance exam, and for proper type of the test, common choices were an objective structured clinical examination (27.8%), a scenario-based nursing performance examination (37.4%), and a clinical performance examination using standardized patients (32.3%). When multiple answers were allowed, the important abilities that many thought needed to be evaluated were basic nursing skills (83.2%), critical thinking skills (56.4%), communication skills (55.9%), and clinical reasoning skills (54.5%).

Under the assumption that a performance exam will be developed and implemented, many chose across-the-board administration of the test managed by a central institution (the Korea Health Personnel Licensing Examination Institute) as the most preferable form of test management (54.9%). Regarding the proper timing for the test, ‘after the written test’ received the most votes (49.5%), but ‘all year round’ also received many votes (32.3%) (Table 3).

General characteristics of the participants: The participants in the FGIs consisted of 28 professors, including clinical instructors and nurses at hospitals. Their ages ranged from 32 to 55, clinical experience from 2 to 30 years, and teaching career from 3 to 10 years; overall, the participants showed considerable variation in their experiences.

Main results: Both supporting and opposing views on the implementation of a performance exam were articulated, and the main results are presented in Table 4.

(1) Views supporting the implementation of clinical performance exam

Regarding the expected effects of introducing clinical performance exam, there were high hopes about the enhanced clinical and practical abilities of new nurses who will pass the national examination. The new recruits will have to undergo extensive training while preparing for the performance exam, which will improve their clinical performance more than preparing for the current examination. These improvements in their clinical abilities will then boost their adaptability to clinical situations, which will help reduce the turnover intention of new nurses who have worked for less than a year.

However, schools with practicum courses anticipated that the performance exam would bring about the standardization of nursing practice training, which means that the improvement of the evaluation system through more serious high-stakes testing will lead to the standardization of practice courses. This means that the performance exam will change our current knowledge-oriented curriculum into a more harmonious one that balances knowledge, attitudes, and techniques. Additionally, infrastructure for the performance exam will be established in each educational institution as part of preparing for the exam; thus, the quality of nursing schools in general will be improved. Furthermore, interconnections between educational institutions and clinical practitioners will be strengthened, as ensuring a higher test passing rate will require institutional instructors and leaders of clinical institutions to develop a keener interest in teaching the students. The participants’ statements about the aforementioned points are presented below.

< Improvement of new nurses’ clinical performance abilities>

“If a performance exam is introduced, it seems that new nurses’ proficiency in nursing skills will increase faster than before. I think that your other work-related abilities can be improved only when you have the necessary nursing skills.” (Participant 3, professor, age 43)

< Increasing new nurses’ adaptability to clinical situations>

“The ultimate thing that keeps the students from quitting their job at a hospital, when they’re out of school, is their core nursing skills. I think that maybe we need to evaluate them somehow.” (Participant 13, professor, age 37)

< Standardization of nursing practice training>

“Standardization! Every hospital and every ward has a different protocol, and it’s so prevalent (every hospital has different nursing procedures). I think that getting them (different nursing techniques) standardized might be the most important effect of introducing the performance exam.” (Participant 10, professor, age 45)

<Improving the balance of nursing education: knowledge, attitudes, techniques>

“If a performance exam becomes part of the national exam, then it will be reflected in every subject that needs testing…It would be a game-changer, wouldn’t it? I think it will ultimately reinforce the education on techniques and attitudes.” (Participant 28, chief nurse, age 41)

<Establishing the necessary infrastructure at educational institutions>

“I expect that the infrastructure will be established. If a performance exam is included in the national exam, then schools probably will start investing for that purpose. The system creates the basis for expanding the infrastructure, so to speak.” (Participant 23, chief nurse, age 42)

<Strengthening interconnections between educational institutions and practice institutions>

“If a performance exam becomes part of the national exam and if it’s institutionally necessary, then I think that clinical leaders will change their attitudes towards practice instruction. Wouldn’t they feel (more) responsible?” (Participant 14, chief nurse, age 54)

(2) Views opposing the implementation of a performance exam Some universities already have a graduation qualification system, the purpose of which is to manage a balanced nursing curriculum in which nursing students’ clinical performance, skills, and attitudes are improved. Some respondents also pointed out that the accreditation test for nursing education held by KABONE already evaluates nursing students’ core nursing skills, and therefore could be an alternative to an official performance exam; as such, the test might be pointless. In addition, many were concerned that integrating the test into the national exam would significantly increase the burden on institutional instructors as well as on students; in particular, some faculty members in charge of specific courses are more involved in practical training than other faculty members, meaning that the burden will be concentrated on a few professors. Practical examinations tend to cause more stress than knowledge-oriented examinations, and students may fail the national exam due to a single mistake; the graduating seniors would inevitably feel high levels of stress in such circumstances. The participants’ statements about the aforementioned points are presented below.

< The accreditation test for nursing education is sufficient>

“If we’re talking about 20 really simple nursing skills, universities can make students take the accreditation test. Otherwisse, we can do it with a graduation qualification system.” (Participant 24, professor, age 32)

< Increased burden on educators>

“I think many colleges might be preoccupied with nothing but the national exam because a performance exam can be really burdensome. In fact, the reality of our clinical training is that not many instructors are in charge of the training, and there are few opportunities where students can see and learn from actual patients. Finding a solution to that can be really taxing for universities.” (Participant 18, nurse, age 40)

< Meaninglessness of the performance exam>

“Even the performance exam for advanced practice nurses had a low level of difficulty, and there were so many problems that they had to change the performance exam to essentially be a written test. I doubt that an undergrad-level nursing performance exam is necessary.” (Participant 12, nurse, age 36)

(3) Aspects to consider if a performance exam is introduced The first point to be considered when introducing a performance exam as part of the national exam is that efforts first need to be made to ameliorate the quality of clinical training. When students’ main activity in a clinical training institution is observation instead of handson practice, the students cannot be effectively ready for the performance exam. Therefore, cooperation from clinical training institutions must precede the implementation; they are in charge of most clinical training processes, and they have the responsibility of cultivating the students’ critical thinking skills and good judgment.

Moreover, the national nursing licensing examination is different from other health-related exams in that it has significantly more examinees; the examinees need to be effectively managed so that they will not be at any disadvantage. In addition, the assessment content of the performance exam needs to be standardized, and the evaluators also need to be trained so that their assessments will be considered credible and valid, which is paramount.

< Efforts to ameliorate clinical training>

“The nurse educator’s role is important. We get loads of work from the hospital already, and student practical training on top of that…I wish there was some systematic reward. I think the reward has to be officially institutionalized.” (Participant 8, professor, age 42)

<Fulfilling the necessary conditions for the effective management of examinees>

“There are too many people. Concrete measures must be taken for us to be able to accommodate these people. Facilities, equipment for implementing the performance exam, and the cost: these will be huge problems.” (Participant 22, professor, age 39)

<Standardization of the assessment content of the performance exam>

“Every college has a different environment for student practice. And if every hospital has a different standard for nursing skills, there will be chaos when a performance exam is implemented. What has to come first is the standardization of nursing skills, based on which the assessment content of the performance exam must be standardized.” (Participant 24, professor, age 32)

<Ensuring the credibility and validity of the evaluators of the performance exam>

“Having many examinees requires many evaluators. Every evaluator needs to apply the same criteria, so I think that ensuring the evaluators’ credibility and validity will be the most important task.” (Participant 2, nurse, age 36)

We found that nursing professors and clinical experts were concerned about the limitations of the current written test as a method of evaluating examinees’ abilities, and that they had high expectations for the beneficial effects of introducing a performance exam. In particular, their hopes were highest for the effects of a hands-on performance exam on simulators or standardized patients, which was expected to improve nurses’ basic nursing skills. Still, the number of annual examinees (20,000) would be a serious challenge, as the center-controlled en bloc style was deemed the most preferable form of exam administration. Nevertheless, the respondents believed that a performance exam needs to be implemented after confirmation of its credibility and validity, provided the test is appropriate for evaluating nurses’ abilities and facilitating skill improvement. Therefore, the groundwork for the smooth implementation of the test seems to be the next necessary step; this would include preparatory work such as securing physical resources and building a consortium of educators and hands-on workers. Moreover, the introduction of a performance exam will lead to the construction of infrastructure geared to the test in educational institutions, and will create an opportunity for nursing schools to improve the overall quality of their practical training, as indicated by the effect analysis report of clinical skills assessments for medical doctors [6]. According to that report, the introduction of a performance exam in medical doctors caused positive changes such as increased participation by students in the practical training encouraged by professors; additionally, resources were efficiently utilized and regional consortiums were vitalized, all leading to the improvement of clinical techniques, attitudes to patients, confidence, and communication skills among interns and residents [7]. Another report reported that mandating 4 years of clinical performance examinations for nursing students, in comparison to written tests, increased the chance of evaluating the students’ clinical and communication skills [8]. Considering these reports, it seems evident that continuous assessment and management are the keys for enhancing clinical and communication skills.

Nonetheless, participants articulated other opinions as well, arguing that the quality of education would be guaranteed only if the education stays faithful to the curriculum and if the KABONE accreditation process functions properly. According to a perception survey on the accreditation test for nursing education, professors in charge of basic nursing education thought that the accreditation test was necessary for the physical environment of practical training to be ameliorated [9]. This means that the institutional apparatus plays an important role in creating the basis for practical training. Therefore, it is necessary to analyze how much the practical abilities of nursing students changed after the implementation of the accreditation evaluation (by a meta-evaluation of the accreditation evaluation), and if it turns out that the accreditation system does not guarantee a sustainable education, remedies will have to be made to complement the written test.

Most respondents chose a hands-on performance exam using simulators or standardized patients as the most suitable format for a performance exam as part of the national nursing licensing exam. This seems to reflect the positive outcomes of core nursing skill education reinforced by KABONE, and the perspective that firsthand training is more fitting to the nature of a nursing job than virtual-reality based training. The skill assessment test for medical doctors—which is made up of 12 exam subjects—has 3,500 annual candidates. The performance exam for nurses, in contrast, has roughly 20,000 annual examinees; there must be regional centers that can accommodate these people and increase the total capacity before implementation. Additionally, this study indicates that the performance exam should be held multiple times a year, and if we accept the opinion that examinees should have multiple chances to take the test, it is advisable to maintain the test system on an all-year-round basis.

The exam subjects need to comprise items that evaluate not simple skills, but problem-solving skills in complicated situations; only then will the test be able to accomplish its high-level goals. Additionally, considering the target abilities of the performance exam, the test must be a clinical performance examination requiring a complex approach to standardized patients or simulators. If a part of the purpose of the test is to increase the opportunity to train for the practical skills required on the job, it is desirable to allow a retake when a candidate fails the test.

In preparation for the introduction of a clinical performance exam as part of the national nursing licensing examination, this study examined the necessity of a performance exam and possible action plans for implementation. In conclusion, the current national examination is perceived to be limited in its ability to select nurses who are well-qualified for the changing medical environment, and thus, introducing a clinical performance exam as part of the national nursing licensing examination is expected to improve nurses’ work ability. Many opinions collected in this study simultaneously present hopes, anticipated problems, and possibilities regarding the prerequisites for a performance exam.

The nursing education process must take 2 preparatory steps: improving the quality of clinical practice training, and standardizing the assessment content of a performance exam. To these ends, we suggest establishing a consortium for standardizing the practical training of over 200 nursing schools. Medical schools will be a good reference; they established regional consortiums, through which they cooperatively developed the assessment content of the performance exam before it was implemented. When the test is implemented, a sufficient number of evaluators will have to be secured and trained, and plans for the effective management of examinees will be necessary. The Korea Health Personnel Licensing Examination Institute needs a taskforce to prepare for the year-round administration of the test.