The program evaluation is generally classified into three types: planning, summative, and formative evaluation. Even though distinctions between the different types of evaluation are sometimes blurred, differentiating their intent helps us in clarifying our understanding of evaluation process (1). Planning evaluation takes place before a program begins to give those involved in program development a precise understanding of the program, and it is sometimes referred to as "pre-formative evaluation" (2). Summative evaluation is the typical and most common type of evaluation, which is conducted at the end of a program to provide decision-makers with judgements about the program's overall merit or worth. However, although necessary, it often comes too late to be much help (3). On the other hand, formative evaluation occurs during the process of a program to provide those who are responsible ongoing information about whether things are going as planned and whether expected progress is being made. If not, this same information can be used to guide necessary improvements, before it is too late (4). Even though there is a great extent of literature on formative evaluation, most of them mainly focus on its conceptual framework, methodology and use. Surprisingly, the subsequent effect of using the findings of formative evaluation has not received systematic attention, and few researches demonstrate this by comparing data from the initial program with the final program to show whether there was an improvement in program implementation and impacts (5). This study aimed to evaluate the subsequent effect of the formative program evaluation based on a case study of a clinical training program in Lao People's Democratic Republic (PDR).

The main health care delivery system of Lao PDR is a government-controlled, public system which has a strong vertical structure with three levels: central, provincial, and district level (6). Even though all health care professionals in the country are required by law to continuously improve their knowledge and skills, the continuing professional development (CPD) training system in Lao PDR has not yet been functioning well (7). In 2012, under the support of Korea International Cooperation Agency (KOICA), faculties from Seoul National University (SNU) College of Medicine in Korea and University of Health Sciences (UHS) in Lao PDR launched the "Continuing Professional development training project to strengthen the capacity of provincial and district hospitals in Lao PDR" (8). At the beginning of the project, UHS faculties conducted a needs assessment survey in Luang Prabang province, which was the pilot area for this project. Based on the needs assessment results, Korean and Lao faculties worked together to develop a CPD training program and a handbook including 80 clinical topics on 5 major clinical specialties: internal medicine, surgery, obstetrics and gynecology, pediatrics, and emergency medicine. And then, UHS faculties provided a 1-week 'training of trainers' program for 30 medical faculties of the Luang Prabang provincial hospital. Also, the training management committee composed of provincial hospital executives and department chairs was established to monitor and manage the training effectively. Finally, the trained provincial hospital faculties provided a 10-week training program in the provincial hospital for the district hospital health professionals in Luang Prabang province. The training was conducted 4 times consecutively over 2 yr, and 12 medical professionals, 1 from each district hospital, were invited to each training program. Among the total 48 medical professionals, 35 of them were medical assistants, and the other 13 were medical doctors. The training was composed of 5 major clinical sections, and the trainees rotated each clinical section every 2 weeks. Lecture about clinical topics, observation of trainers' performance, and trainee's own medical practice with trainers' feedback were the main training methods.

Formative program evaluation was conducted during the whole process to continuously improve the training program (Fig. 1). Kirkpatrick model was applied for the program evaluation covering level 1 (reaction) and level 3 (transfer) (9). Questionnaire survey and focus group interviews with the trainees were used to evaluate the reaction of the trainees. The questionnaire was designed by the co-work of faculties from SNU and UHS based on the literature review (1011). It was composed of 14 items, regarding the goals and objectives, relevance, organization, trainers' knowledge, trainers' communication, trainers' preparedness, training methods, discussion and interaction, practical session, handbook, facilities, schedule, applicability, and helpfulness of the training program. The trainees were asked to rate the items with a 5-point Likert scale (1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree) (Table 1). The survey was conducted every two weeks at the end of each clinical section of the training program. Focus group interview with the trainees was facilitated by the UHS faculties at the end of each 10-week training program.

The transfer of the trainees was evaluated through the review of medical records written by the trainees. Total 354 medical records written by 44 trainees before and 3 months after the training program were collected. Global rating with a 5-point anchored scale (1, novice; 2, advanced beginner; 3, intermediate; 4, proficient; 5, expert) was used for the evaluation. A total of 25 trainers received a half-day workshop before assessing the medical records, and they practiced with some pilot medical records and received feedback during the workshop. The medical records were coded by the coordinator, and they were randomly distributed to the trainers without any personal and time information. The medical records were assessed twice from a different trainer. The difference of average scores between the medical records written before the training and those written after the training was calculated to evaluate the transfer of the trainees.

The evaluation data was collected under the responsibility of the training management committee and analyzed by the study team. The results were shared with the committee and the trainers after the end of each batch of the training program. The committee and the trainers reached a consensus on how to improve the training program through the group discussion based on the evaluation results. After the final end of the training program, the average scores of the survey results and the achievement of global rating scores of medical records among the four batches were analyzed statistically with the ANOVA using SPSS.

According to the results of the survey, the trainees were quite satisfied with the training program (Table 2). However, especially in the early period of the training program, the trainees were not so satisfied in such areas as the organization and schedule of the training program, and discussion, interaction, and practical sessions during the training program. From the second batch, there was continuous increase of the satisfaction of the trainees in all the items of the questionnaire, and most of the significant improvement happened between the first and the second batch (Supplementary Fig. 1).

The focus group interviews provided similar but more in-depth view of the trainees' reactions (Table 3). Most of the comments about the training program were positive. However, especially at the first batch of the training, some trainees suggested that there should be more practice and interaction in the training program and the trainers should pay more attention and provide more explanation to the trainees. From the second batch of the training, comments about the program and the trainers became more positive, and there was less criticism about the issues that were previously mentioned. In the second batch, some trainees suggested that the training should be more applicable to the situation of district hospitals, which also seemed to be improved from the third batch of the training program.

The average global rating scores of the medical records which were written after the training program were higher than those written before the training, except the second batch (Table 4). And the achievement of the average global rating scores between the medical records before the training and those after the training increased from the first batch to the last batch, especially between the second and the third batch (Supplementary Fig. 2).

The purpose of this study was to evaluate the effect of the formative program evaluation, which was applied to continuously improve the training program. According to the evaluation results, there was continuous improvement of the reaction and the transfer of the trainees from the first batch to the last batch of the training program. We might have a close look at the ways how the formative program evaluation contributed to the continuous improvement of the training program.

At the first batch of the training program, the trainees were not so satisfied in some areas like organization and schedule of the training program, and discussion, interaction, and practical sessions in the training program. Some trainees thought the training activities were not enough for them, and there should be more practical sessions like bed side teaching or real practice rather than lectures. Trainees also suggested that the trainer should pay more attention to them and there should be more discussion and interaction between the trainer and the trainees. The review of the medical records written by the first batch trainees showed a minimal achievement of global rating scores. These results of the formative program evaluation were shared with the committee and the trainers at the end of the first batch of the training program. The committee and the trainers reached a consensus on how to improve the training program through the group discussion based on the evaluation results. They decided to increase the training activities, especially practical sessions like bed side teaching and real practice under supervision. And even during the lecture time, they agreed to foster more interaction and discussion between the trainers and the trainees. The training management committee also emphasized to the trainers that the trainees are not young students, but experienced medical professionals.

At the second batch of the training program, the reaction of the trainees was much improved than the first batch. The average scores of all the items in the survey were much increased, and there was no item which was rated below 4.0. Also the trainee's comments, especially about the trainers, in the focus group interview became more positive than before. However, the review of the medical records written by the second batch trainees did not show any achievement of global rating scores. After a group discussion based on the results of the survey and focus interview, the training management committee and the trainers reached on a consensus that there still should be more practical sessions for the trainees, which later could be applicable to the medical practice in the district hospitals.

At the third batch of the training program, the trainees were more satisfied with the practical sessions and more trainees appreciated the applicability of the training program. And the achievement of global rating scores of the medical records was much increased comparing to the first and the second batch. The program evaluation results of the fourth batch showed a minimal improvement of the reaction and transfer of the trainees.

There were several key factors which should be noted in this study on the formative program evaluation. First of all, two levels, level 1 (reaction) and level 3 (transfer), of the Kirkpatrick model were selected for the program evaluation for the reason that the reaction of the trainees should be closely related to the implementation of the training program and the transfer of the trainees could be a reliable parameter of the outcome or impact of the training program (12). Second, the survey for the evaluation of the trainees' reaction was conducted every two weeks during the training program to improve the reliability of the ratings. It was to minimize the dominating effect of the trainee's emotional experiences of both the peak and the end of the training program (13). Third, focus group interviews were added to seek more deeply the problems of the training program which were discovered by the survey results. By this, the quantitative evaluation using a Likert scale and qualitative evaluation using focus group interviews, both of which have their relative merits (14), were integrated into the formative program evaluation process. Finally, to evaluate the transfer of the trainees, medical records which were written 3 months after the training program were collected, analyzed, and compared with those written before the training. It was based on the evidence from the previous studies that most meaningful changes would take place over a longer period of time than just right after the training (1516).

There were some limitations of this study, too. We were not able to assign a control group and conduct a case-control study due to the small size of the trainees group. And the formative program evaluation did not cover the level 2 (learning) and level 4 (result) of the Kirkpatrick model because of the feasibility and the time limit of the evaluation process. Further study is needed for more delicate study design and concrete results. However, even though it was a pilot clinical training program in Lao PDR, the plan for the formative program evaluation was designed from the beginning of the project, and was shared with the all the participants. The whole evaluation process was conducted by the Lao health professionals under the guidance and support of the study team.

In conclusion, the results showed that the formative program evaluation contributed to the continuous improvement of the training program.