PDF
ePub
Citation
Print
I. Introduction
The National Health Insurance Corporation Ilsan Hospital (NHIIH) was established by the National Health Insurance Corporation in Korea as a reference hospital for health policy on insurance rates and hospital costs. NHIIH opened in March, 2000 with the size of 745 beds. It is located in the northwest part of Gyeonggi-do, and there are 55 centers and clinics including a cardiovascular center, emergency center, artificial joint center, hyperlipidemia clinic and arrhythmia clinic. There are 24 clinical departments with 1,400 medical specialists. In 2011, NHIIH treated 781,323 outpatients and 29,424 inpatients. NHIIH introduced a computerized provider order entry (CPOE) system, admission/discharge/transfer (ADT) system, and laboratory information system (LIS) in 2000. It introduced a picture archive and communication system (PACS) and data warehouse (DW) system in 2001. Based on these, NHIIH introduced the new integrated hospital information system with the focus on the Electronic Medical Record (EMR) on November 4, 2011.
To define the goals and scope of a new system, the information strategy plan (ISP) was developed in December 2009. An organization for developing the new system was established with the EMR task force comprised of the following three teams in July 2011: business plan team, contents development team, and supporting team. The business plan team was responsible for analyzing and redesigning the business processes in order to improve the hospital's operating efficiency. It was comprised of medical, nursing, medical supporting, and administrative staffs. The contents development team was responsible for developing quality indicators, standards, codes, and user interface guidelines. The supporting team was responsible for management support and system development support. During the development period, 573 interviews were conducted to identify user requirements, two workshops were organized to monitor development activities, and 175 user training sessions were conducted to train key users.
The performance of the new system was evaluated from both qualitative and quantitative perspectives before and after the implementation in order to improve user satisfaction and justify the investment on the system. The qualitative evaluation dealt with factors related to user satisfaction and improvement in the business process based on the performance reference model (PRM) which was developed by the Office for Management of Budget (OMB) in the United States [1]. PRM is a standardized framework to measure the performance of major information systems (IS) investments and their contribution to program performance. PRM has been widely used in evaluating IS performance for government projects in the United States. Chung et al. [2] applied PRM to evaluate IS performance for several e-government projects in Korea. Since PRM evaluates IS based on the key performance indicators (KPIs) for each of three system layers, input, process, and business, it provides a comprehensive view of the overall IS performance. Chung et al. [2] also developed PRM version 2.0 as a standard performance evaluation tool and identified 138 KPIs for evaluating the performance of government IS projects. However, many of these KPIs are not applicable in healthcare settings because they were primarily developed for IS's for government projects unrelated to healthcare. Moreover, there are not many studies that compared the changes in KPIs before and after the introduction of IS. Therefore, there is a need to analyze KPIs for evaluating such performance of IS projects in a healthcare setting based on the PRM framework.
The quantitative evaluation primarily deals with the economic effects of IS. In order to analyze the economic effects of the new system, waiting times at the registration desk and outpatient departments were measured, and business processes were compared before and after the introduction of new system. Although several studies have evaluated economic benefits of information systems projects in healthcare [3,4], most of traditional cost-benefits analyses did not adequately account for intangible or indirect benefits such as savings from the reduction in medication errors. Since hospital information systems (HIS) tend to have a great deal of intangible benefits, there is a need for another approach to account for them. Parker et al. [5] introduced the information economics (IE) approach to account for intangible benefits by systematically analyzing the intangible benefit items. Chae et al. [6,7] modified them to the healthcare setting and used to analyze the economic effects of CPOE and EMR. Kang et al. [8] also used the IE approach to analyze the economic effects of EMR, but they mostly relied on the hypothesized performance indicators in their analysis.
The purposes of this paper were to evaluate the qualitative performance of the new information systems at NHIIH based on PRM and compared the waiting time before and after the introduction of new system at NHIIH. Based on these results, an economic analysis was conducted by using the IE approach.
II. Methods
1. Data Collection
Questionnaire surveys for qualitative evaluation were conducted three times to identify all factors influencing system success: three months before (August 8, 2011) and after three (February 4, 2012) and seven months (June 4, 2012) of the introduction of the new system implemented on Nov. 4, 2011. On each survey, 266, 314, and 329 users responded. Interns and residents were excluded in all surveys because they often rotated to other hospitals during their training.
The waiting time for registration and business process of 475 patients was measured three months before the system's introduction (August 8 and 9, 2011), and 700 patients seven months after that (June 4 and 5, 2012) at the central registration desk; and 200 patients at two internal medicine outpatient departments (pulmonology and gastroenterology) three months before the system's introduction and 300 patients seven months after the introduction.
2. Qualitative Performance Evaluation
Twenty three KPIs were derived from three layers based in PRM. Specifically, the input layer was comprised of IT factors (e.g., system quality, information quality, user understanding of system, information sharing, and security check) and human factors (top management support). The process layer was comprised of the business transaction process (efficiency of process and quality of business transaction) and the management process (regulation management, knowledge management, and security management). The business layer was comprised of business aspects (effectiveness in business) and the user aspects (user satisfaction) [9]. Each KPI satisfaction score was measured for four types of users (i.e., doctors, nurses, medical support staffs, and administrative staffs) in 3-point scales (1, not satisfied; 2, in-between; and 3, satisfied).
3. Economic Analysis
The economic performance of HIS was evaluated by comparing costs and benefits. Cost items include hardware, database, software, system infrastructure, and system audit. Maintenance cost was zero for the first year. The scope of the benefits was broadened to values based on the IE approach in order to account for the intangible benefits. The IE approach categorizes them into 5 specific value items: direct benefits, value acceleration, value linkage, value restructuring, and innovation valuation. Direct benefits refer to the traditional benefits from IS, such as savings in personnel costs due to reduced workloads. Value acceleration refers to the value associated with time factors, such as the reduced opportunity costs for patients due to the reduced waiting time. Value linkage refers to the value accrued from compounded effects, such as savings for hospital costs due to reduced medication errors. Value restructuring refers to the value accrued from increased productivity by reengineering the business process.
III. Results
1. Characteristics of the Subjects
Characteristics of the subjects of three surveys were described in Table 1. In each survey, over 70% of the subjects were female as more than 50% of the subjects were nurses. Almost half of the subjects were in their age 30s; younger subjects (aged 20-29) participated less than the older subjects (aged 40 and over) in the first survey, however, they participated in the second and third surveys more than the older subjects. Doctors and nurses, who were the key users of EMR, were almost 80% of the subjects in all surveys. A total of 32 doctors participated in all three surveys. Over 60% of the subjects had more than seven years of computer experience. Finally, less than 30% of the subjects used the system for 40 hours or less per week in the first survey, but more than 60% of the subjects used the system for 40 hours or less per week in the second and third surveys.
2. Waiting and Business Processing Time Measurement
Both waiting time and registration time at the central registration desk were significantly reduced after the introduction of the system, as seen in Table 2. Waiting and business processing time at the internal medicine department was significantly reduced after the introduction of the system. Treatment time at the same department has also reduced, but the difference was not significant.
3. Qualitative Performance Evaluation
As see in Table 3, all KPI satisfaction scores in the input layer (system quality, information quality, and user understanding of system, information sharing and top management support) except the security check initially decreased after three months of the introduction of the system and increased after seven months. However, both access control and virus protection scores in the security management were continually increased three months prior to the introduction of the system. The changes in the KPI satisfaction scores were all statistically significant except for information maintenance and top management support.
In the process layer, the KPI satisfaction scores for efficiency of the process, quality process, and knowledge management decreased after three months since the system introduction then increased after seven months. These changes in the KPI satisfaction scores were all statistically significant. On the other hand, the KPI satisfaction scores for observing the regulation continually decreased, but for the security management the KPI satisfaction scores increased after three months from the system introduction then decreased after seven months. The changes in the latter two KPIs were not statistically significant.
In the business layer, all KPI satisfaction scores showed the same significant pattern in three survey periods. They decreased after three months since the inception of the system, and increased after seven months.
4. Economic Performance Evaluation
Cost items include hardware, database, software, system infrastructure, and system audit. Maintenance cost was zero for the first year. The total cost was KRW 8,529,570,620 (about USD 7,107,975), as seen in Table 4.
Direct benefits were savings from the reduction in outpatient preparation time, chart delivery time, chart storage costs, and supply costs. These amounted to KRW 1,660,540,000 and the benefit to cost (B/C) ratio was 0.019. The traditional cost-benefit analysis accounting for the direct benefits alone could not justify the costs. Value acceleration was the value associated with the time factor, such as the revenue increase due to the reduction in patient waiting time and the savings from the reduction in nursing recording time. These amounted to KRW 1,446,021,416, and the B/C ratio of the cumulative values increased to 0.36. Value linkage was the value accrued from all compounded effects, such as the savings from the reduction in medication errors. Medication errors obtained by multiplying the total amount of drug expenses by the reduction rate in medication errors were significantly reduced after the introduction of the system resulting in the highest increase in values which was. These amounted to KRW 6,289,827,131 and the B/C ratio of the cumulative values increased to 1.10. Finally, value restructuring was the value accrued from the increased efficiency in business process, such as the savings from the reduction in returned narcotic drugs, accounting processing time, chart management time, and supplies input time. These amounted to KRW 70,162,072. The final B/C ratio of the cumulative values KRW 9,583,190,820 (about USD 8,599,417) increased to 1.12. Accordingly, the IE analysis showed the economic profitability of the new system if all intangible benefits were included, as seen in Table 5.
IV. Discussion
NHIIH began the implementation of the newly integrated hospital information system on November 4, 2011. Three months prior to its implementation, performance of the old information system was qualitatively evaluated based on PRM in order to identify problematic areas for the old system and collect the baseline data to compare with the performance of the new system. All KPI satisfaction scores in the input layer (system quality, information quality, and recognition of user's information, information share and top management support) except the security check initially decreased after three months of the introduction of the system and increased after seven months. This result coincides with the study of Kim and Choi [10]. However, both access control and virus protection scores in the security management continually increased since the introduction of the system. This may be due to the fact that the electronic signatures were required in every record and overall security check for access control were reinforced after the introduction of the new system.
In the process layer, the KPI satisfaction scores for efficiency of the process, quality process, and knowledge management decreased after three months since the system introduction then increased after seven months of the system implementation. On the other hand, the satisfaction scores for observing the regulation continually decreased perhaps due to the confusion in the process of adjusting to the new working process. In the business layer, all KPI scores showed the same significant pattern in three surveys. They decreased after the first three months and increased after seven months.
Economic analysis was performed for the newly developed HIS by using the IE approach in order to systematically quantify the intangible benefits. IE attempts to categorize the benefits into the following five value types: direct benefits, value acceleration, value linkage, value restructuring, and innovation valuation. While actual hospital data were used in most of the analysis, some data were extrapolated from the literature if actual data were not available. The patient waiting time for two outpatient departments were compared before and after the introduction of the system in order to estimate the time savings from the reduction in waiting time. The B/C ratio was 0.02 if only the direct benefit was included, but increased to 0.36 with the addition of value acceleration, to 1.10 with value linkage, and to 1.12 with value restructuring. Medication errors were significantly reduced after the introduction of the system resulting in the highest increase in values. The previous study on the economic effects of hospital information systems shows similar findings. Bates et al. [11] reported that CPOE and clinical decision support system reduced medication error rates by 55%. In a similar study, Kaushal et al. [3] reported that CPOE saved a total of 28.5 million dollars during 11 years of its operation at the Brigham and Women's Hospital. The largest cumulative savings were from renal dosing guidance (USD 6.3 million), improved nursing time utilization (USD 6.0 million), specific drug guidance (USD 4.9 million), and adverse drug events (ADE) prevention (USD 3.7 million). Wu et al. [4] found the similar savings from the reduction in ADEs at the Toronto General Hospital. They estimated that 261 ADEs were prevented resulting in USD 3,322,000 over the 10-year time horizon. Girosi et al. [12] also found that hospitals with more than 100 beds can save 1 billion dollars annually by reducing the 20,000 side effects of drug. The IE analysis demonstrated the economic profitability of the new system when all intangible benefits were included.
There are limitations with the qualitative and quantitative evaluation. First, many users at NHIIS were still not satisfied with the system because some user requirements were not fully reflected to the system there were frequent changes in the program. Second, some medication error data were based on the limited study of economic evaluation at one hospital so that the results are not generalizable to other institutions. Finally, our maintenance cost was zero and this resulted in lower implementation costs. Implementation costs could vary depending on the type of institution.
In the future, performances of the new system should be evaluated again after the system is more stabilized. Specifically, the KPIs from the previous survey should be compared with the KPIs from the fourth survey to determine whether the new information system will further improve user satisfaction, especially for medical staffs, and improve efficiency in the business process. In addition, some economic data were based on institutional estimates and this requires further study to improve accuracy in economic evaluation.
XML Download