Journal List > Healthc Inform Res > v.18(3) > 1075615

Han, Chae, Boo, Kim, Yoon, and Kim: Performance Analysis of Hospital Information System of the National Health Insurance Corporation Ilsan Hospital

Abstract

Objectives

This study evaluated the qualitative and quantitative performances of the newly developed information system which was implemented on November 4, 2011 at the National Health Insurance Corporation Ilsan Hospital.

Methods

Registration waiting time and changes in the satisfaction scores for the key performance indicators (KPI) before and after the introduction of the system were compared; and the economic effects of the system were analyzed by using the information economics approach.

Results

After the introduction of the system, the waiting time for registration was reduced by 20%, and the waiting time at the internal medicine department was reduced by 15%. The benefit-to-cost ratio was increased to 1.34 when all intangible benefits were included in the economic analysis.

Conclusions

The economic impact and target satisfaction rates increased due to the introduction of the new system. The results were proven by the quantitative and qualitative analyses carried out in this study. This study was conducted only seven months after the introduction of the system. As such, a follow-up study should be carried out in the future when the system stabilizes.

I. Introduction

The National Health Insurance Corporation Ilsan Hospital (NHIIH) was established by the National Health Insurance Corporation in Korea as a reference hospital for health policy on insurance rates and hospital costs. NHIIH opened in March, 2000 with the size of 745 beds. It is located in the northwest part of Gyeonggi-do, and there are 55 centers and clinics including a cardiovascular center, emergency center, artificial joint center, hyperlipidemia clinic and arrhythmia clinic. There are 24 clinical departments with 1,400 medical specialists. In 2011, NHIIH treated 781,323 outpatients and 29,424 inpatients. NHIIH introduced a computerized provider order entry (CPOE) system, admission/discharge/transfer (ADT) system, and laboratory information system (LIS) in 2000. It introduced a picture archive and communication system (PACS) and data warehouse (DW) system in 2001. Based on these, NHIIH introduced the new integrated hospital information system with the focus on the Electronic Medical Record (EMR) on November 4, 2011.
To define the goals and scope of a new system, the information strategy plan (ISP) was developed in December 2009. An organization for developing the new system was established with the EMR task force comprised of the following three teams in July 2011: business plan team, contents development team, and supporting team. The business plan team was responsible for analyzing and redesigning the business processes in order to improve the hospital's operating efficiency. It was comprised of medical, nursing, medical supporting, and administrative staffs. The contents development team was responsible for developing quality indicators, standards, codes, and user interface guidelines. The supporting team was responsible for management support and system development support. During the development period, 573 interviews were conducted to identify user requirements, two workshops were organized to monitor development activities, and 175 user training sessions were conducted to train key users.
The performance of the new system was evaluated from both qualitative and quantitative perspectives before and after the implementation in order to improve user satisfaction and justify the investment on the system. The qualitative evaluation dealt with factors related to user satisfaction and improvement in the business process based on the performance reference model (PRM) which was developed by the Office for Management of Budget (OMB) in the United States [1]. PRM is a standardized framework to measure the performance of major information systems (IS) investments and their contribution to program performance. PRM has been widely used in evaluating IS performance for government projects in the United States. Chung et al. [2] applied PRM to evaluate IS performance for several e-government projects in Korea. Since PRM evaluates IS based on the key performance indicators (KPIs) for each of three system layers, input, process, and business, it provides a comprehensive view of the overall IS performance. Chung et al. [2] also developed PRM version 2.0 as a standard performance evaluation tool and identified 138 KPIs for evaluating the performance of government IS projects. However, many of these KPIs are not applicable in healthcare settings because they were primarily developed for IS's for government projects unrelated to healthcare. Moreover, there are not many studies that compared the changes in KPIs before and after the introduction of IS. Therefore, there is a need to analyze KPIs for evaluating such performance of IS projects in a healthcare setting based on the PRM framework.
The quantitative evaluation primarily deals with the economic effects of IS. In order to analyze the economic effects of the new system, waiting times at the registration desk and outpatient departments were measured, and business processes were compared before and after the introduction of new system. Although several studies have evaluated economic benefits of information systems projects in healthcare [3,4], most of traditional cost-benefits analyses did not adequately account for intangible or indirect benefits such as savings from the reduction in medication errors. Since hospital information systems (HIS) tend to have a great deal of intangible benefits, there is a need for another approach to account for them. Parker et al. [5] introduced the information economics (IE) approach to account for intangible benefits by systematically analyzing the intangible benefit items. Chae et al. [6,7] modified them to the healthcare setting and used to analyze the economic effects of CPOE and EMR. Kang et al. [8] also used the IE approach to analyze the economic effects of EMR, but they mostly relied on the hypothesized performance indicators in their analysis.
The purposes of this paper were to evaluate the qualitative performance of the new information systems at NHIIH based on PRM and compared the waiting time before and after the introduction of new system at NHIIH. Based on these results, an economic analysis was conducted by using the IE approach.

II. Methods

1. Data Collection

Questionnaire surveys for qualitative evaluation were conducted three times to identify all factors influencing system success: three months before (August 8, 2011) and after three (February 4, 2012) and seven months (June 4, 2012) of the introduction of the new system implemented on Nov. 4, 2011. On each survey, 266, 314, and 329 users responded. Interns and residents were excluded in all surveys because they often rotated to other hospitals during their training.
The waiting time for registration and business process of 475 patients was measured three months before the system's introduction (August 8 and 9, 2011), and 700 patients seven months after that (June 4 and 5, 2012) at the central registration desk; and 200 patients at two internal medicine outpatient departments (pulmonology and gastroenterology) three months before the system's introduction and 300 patients seven months after the introduction.

2. Qualitative Performance Evaluation

Twenty three KPIs were derived from three layers based in PRM. Specifically, the input layer was comprised of IT factors (e.g., system quality, information quality, user understanding of system, information sharing, and security check) and human factors (top management support). The process layer was comprised of the business transaction process (efficiency of process and quality of business transaction) and the management process (regulation management, knowledge management, and security management). The business layer was comprised of business aspects (effectiveness in business) and the user aspects (user satisfaction) [9]. Each KPI satisfaction score was measured for four types of users (i.e., doctors, nurses, medical support staffs, and administrative staffs) in 3-point scales (1, not satisfied; 2, in-between; and 3, satisfied).

3. Economic Analysis

The economic performance of HIS was evaluated by comparing costs and benefits. Cost items include hardware, database, software, system infrastructure, and system audit. Maintenance cost was zero for the first year. The scope of the benefits was broadened to values based on the IE approach in order to account for the intangible benefits. The IE approach categorizes them into 5 specific value items: direct benefits, value acceleration, value linkage, value restructuring, and innovation valuation. Direct benefits refer to the traditional benefits from IS, such as savings in personnel costs due to reduced workloads. Value acceleration refers to the value associated with time factors, such as the reduced opportunity costs for patients due to the reduced waiting time. Value linkage refers to the value accrued from compounded effects, such as savings for hospital costs due to reduced medication errors. Value restructuring refers to the value accrued from increased productivity by reengineering the business process.

III. Results

1. Characteristics of the Subjects

Characteristics of the subjects of three surveys were described in Table 1. In each survey, over 70% of the subjects were female as more than 50% of the subjects were nurses. Almost half of the subjects were in their age 30s; younger subjects (aged 20-29) participated less than the older subjects (aged 40 and over) in the first survey, however, they participated in the second and third surveys more than the older subjects. Doctors and nurses, who were the key users of EMR, were almost 80% of the subjects in all surveys. A total of 32 doctors participated in all three surveys. Over 60% of the subjects had more than seven years of computer experience. Finally, less than 30% of the subjects used the system for 40 hours or less per week in the first survey, but more than 60% of the subjects used the system for 40 hours or less per week in the second and third surveys.

2. Waiting and Business Processing Time Measurement

Both waiting time and registration time at the central registration desk were significantly reduced after the introduction of the system, as seen in Table 2. Waiting and business processing time at the internal medicine department was significantly reduced after the introduction of the system. Treatment time at the same department has also reduced, but the difference was not significant.

3. Qualitative Performance Evaluation

As see in Table 3, all KPI satisfaction scores in the input layer (system quality, information quality, and user understanding of system, information sharing and top management support) except the security check initially decreased after three months of the introduction of the system and increased after seven months. However, both access control and virus protection scores in the security management were continually increased three months prior to the introduction of the system. The changes in the KPI satisfaction scores were all statistically significant except for information maintenance and top management support.
In the process layer, the KPI satisfaction scores for efficiency of the process, quality process, and knowledge management decreased after three months since the system introduction then increased after seven months. These changes in the KPI satisfaction scores were all statistically significant. On the other hand, the KPI satisfaction scores for observing the regulation continually decreased, but for the security management the KPI satisfaction scores increased after three months from the system introduction then decreased after seven months. The changes in the latter two KPIs were not statistically significant.
In the business layer, all KPI satisfaction scores showed the same significant pattern in three survey periods. They decreased after three months since the inception of the system, and increased after seven months.

4. Economic Performance Evaluation

Cost items include hardware, database, software, system infrastructure, and system audit. Maintenance cost was zero for the first year. The total cost was KRW 8,529,570,620 (about USD 7,107,975), as seen in Table 4.
Direct benefits were savings from the reduction in outpatient preparation time, chart delivery time, chart storage costs, and supply costs. These amounted to KRW 1,660,540,000 and the benefit to cost (B/C) ratio was 0.019. The traditional cost-benefit analysis accounting for the direct benefits alone could not justify the costs. Value acceleration was the value associated with the time factor, such as the revenue increase due to the reduction in patient waiting time and the savings from the reduction in nursing recording time. These amounted to KRW 1,446,021,416, and the B/C ratio of the cumulative values increased to 0.36. Value linkage was the value accrued from all compounded effects, such as the savings from the reduction in medication errors. Medication errors obtained by multiplying the total amount of drug expenses by the reduction rate in medication errors were significantly reduced after the introduction of the system resulting in the highest increase in values which was. These amounted to KRW 6,289,827,131 and the B/C ratio of the cumulative values increased to 1.10. Finally, value restructuring was the value accrued from the increased efficiency in business process, such as the savings from the reduction in returned narcotic drugs, accounting processing time, chart management time, and supplies input time. These amounted to KRW 70,162,072. The final B/C ratio of the cumulative values KRW 9,583,190,820 (about USD 8,599,417) increased to 1.12. Accordingly, the IE analysis showed the economic profitability of the new system if all intangible benefits were included, as seen in Table 5.

IV. Discussion

NHIIH began the implementation of the newly integrated hospital information system on November 4, 2011. Three months prior to its implementation, performance of the old information system was qualitatively evaluated based on PRM in order to identify problematic areas for the old system and collect the baseline data to compare with the performance of the new system. All KPI satisfaction scores in the input layer (system quality, information quality, and recognition of user's information, information share and top management support) except the security check initially decreased after three months of the introduction of the system and increased after seven months. This result coincides with the study of Kim and Choi [10]. However, both access control and virus protection scores in the security management continually increased since the introduction of the system. This may be due to the fact that the electronic signatures were required in every record and overall security check for access control were reinforced after the introduction of the new system.
In the process layer, the KPI satisfaction scores for efficiency of the process, quality process, and knowledge management decreased after three months since the system introduction then increased after seven months of the system implementation. On the other hand, the satisfaction scores for observing the regulation continually decreased perhaps due to the confusion in the process of adjusting to the new working process. In the business layer, all KPI scores showed the same significant pattern in three surveys. They decreased after the first three months and increased after seven months.
Economic analysis was performed for the newly developed HIS by using the IE approach in order to systematically quantify the intangible benefits. IE attempts to categorize the benefits into the following five value types: direct benefits, value acceleration, value linkage, value restructuring, and innovation valuation. While actual hospital data were used in most of the analysis, some data were extrapolated from the literature if actual data were not available. The patient waiting time for two outpatient departments were compared before and after the introduction of the system in order to estimate the time savings from the reduction in waiting time. The B/C ratio was 0.02 if only the direct benefit was included, but increased to 0.36 with the addition of value acceleration, to 1.10 with value linkage, and to 1.12 with value restructuring. Medication errors were significantly reduced after the introduction of the system resulting in the highest increase in values. The previous study on the economic effects of hospital information systems shows similar findings. Bates et al. [11] reported that CPOE and clinical decision support system reduced medication error rates by 55%. In a similar study, Kaushal et al. [3] reported that CPOE saved a total of 28.5 million dollars during 11 years of its operation at the Brigham and Women's Hospital. The largest cumulative savings were from renal dosing guidance (USD 6.3 million), improved nursing time utilization (USD 6.0 million), specific drug guidance (USD 4.9 million), and adverse drug events (ADE) prevention (USD 3.7 million). Wu et al. [4] found the similar savings from the reduction in ADEs at the Toronto General Hospital. They estimated that 261 ADEs were prevented resulting in USD 3,322,000 over the 10-year time horizon. Girosi et al. [12] also found that hospitals with more than 100 beds can save 1 billion dollars annually by reducing the 20,000 side effects of drug. The IE analysis demonstrated the economic profitability of the new system when all intangible benefits were included.
There are limitations with the qualitative and quantitative evaluation. First, many users at NHIIS were still not satisfied with the system because some user requirements were not fully reflected to the system there were frequent changes in the program. Second, some medication error data were based on the limited study of economic evaluation at one hospital so that the results are not generalizable to other institutions. Finally, our maintenance cost was zero and this resulted in lower implementation costs. Implementation costs could vary depending on the type of institution.
In the future, performances of the new system should be evaluated again after the system is more stabilized. Specifically, the KPIs from the previous survey should be compared with the KPIs from the fourth survey to determine whether the new information system will further improve user satisfaction, especially for medical staffs, and improve efficiency in the business process. In addition, some economic data were based on institutional estimates and this requires further study to improve accuracy in economic evaluation.

Figures and Tables

Table 1
Characteristics of the users
hir-18-208-i001

Values are presented as number (%).

aFirst survey was conducted three months before the system introduction. bSecond survey was conducted three months after the system introduction. cThird survey was conducted seven months after the system introduction.

Table 2
Outpatient waiting and business processing time
hir-18-208-i002

ap < 0.01.

Table 3
Key Performance Indicator (KPI) satisfaction scores by layers
hir-18-208-i003

ap < 0.01, bp < 0.05.

Table 4
Cost summary of hospital information system implementation
hir-18-208-i004
Table 5
Benefit summary of HIS implementation
hir-18-208-i005

aCumulative benefits/costs ratio. bOpen card system is a post-payment system by using the pre-registered credit card.

Notes

No potential conflict of interest relevant to this article was reported.

Acknowledgments

This article is supported by the grant from the National Health Insurance Corporation Ilsan Hospital. The authors wish to express sincere gratitude to those who responded to surveys and to provide information for the economic analysis.

References

1. Federal Enterprise Architecture. FEA consolidated reference model document version 2.3. 2007. Washington, DC: The Office of Management and Budget.
2. Chung KH, Moon JW, Moon MJ, Kim HS. Methodology for performance evaluation of public information systems. 2008. Gwacheon: Korea Information Society Development Institute.
3. Kaushal R, Jha AK, Franz C, Glaser J, Shetty KD, Jaggi T, et al. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc. 2006. 13(3):261–266.
crossref
4. Wu RC, Laporte A, Ungar WJ. Cost-effectiveness of an electronic medication ordering and administration system in reducing adverse drug events. J Eval Clin Pract. 2007. 13(3):440–448.
crossref
5. Parker MM, Trainor HE, Benson RJ. Information strategy and economics: linking information systems strategy to business performance. 1989. Englewood Cliffs (NJ): Prentice Hall.
6. Chae YM, Lee HJ, Park CR. Economic analysis of order communication system for hospitals. Korean J Prev Med. 1991. 24(4):473–484.
7. Chae YM, Kim S, Cho KW, Kim HS, Kang SH, Kim HH, et al. Economic analysis of hospital information system by information economic approach. J Korean Soc Health Inf Health Stat. 2010. 35(2):195–205.
8. Kang SH, Chung YI, Chae YM. Economic analysis of electronic patient record. J Korean Soc Med Inform. 1997. 3(1):37–47.
crossref
9. Chae YM, Cho KW, Kim HS, Park CB. Evaluation of hospital information system based on the performance reference model. J Korean Soc Health Serv Manag. 2011. 5(1):1–13.
crossref
10. Kim JI, Choi JY. Effectiveness for hospitals management of electronic medical record. Health Welf. 2009. 11(1):87–93.
crossref
11. Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998. 280(15):1311–1316.
crossref
12. Girosi F, Meili RC, Scoville R. Extrapolating evidence of health information technology savings and costs. 2005. Santa Monica: RAND Corporation;Report no.: MG-410-HLTH.
TOOLS
Similar articles