Performance Evaluation of Public Hospital Information Systems by the Information System Success Model

Kyoung Won Cho; Sung-Kwon Bae; Ji-Hye Ryu; Kyeong Na Kim; Chang-Ho An; Young Moon Chae

doi:10.4258/hir.2015.21.1.43

Abstract

Objectives

This study was to evaluate the performance of the newly developed information system (IS) implemented on July 1, 2014 at three public hospitals in Korea.

Methods

User satisfaction scores of twelve key performance indicators of six IS success factors based on the DeLone and McLean IS Success Model were utilized to evaluate IS performance before and after the newly developed system was introduced.

Results

All scores increased after system introduction except for the completeness of medical records and impact on the clinical environment. The relationships among six IS factors were also analyzed to identify the important factors influencing three IS success factors (Intention to Use, User Satisfaction, and Net Benefits). All relationships were significant except for the relationships among Service Quality, Intention to Use, and Net Benefits.

Conclusions

The results suggest that hospitals should not only focus on systems and information quality; rather, they should also continuously improve service quality to improve user satisfaction and eventually reach full the potential of IS performance.

I. Introduction

The Korean health delivery system is heavily dominated by the private sector. The facilities and manpower of most public hospitals are less competitive than those of private hospitals. The adoption rates of Electronic Medical Record (EMR) systems in Korean hospitals in 2012 were the following: 87.5% for tertiary hospitals, 71.7% for general hospitals, and 72.9% for hospitals [1]. However, the adoption rates of public hospitals were much lower than those of private hospitals [2]. As public hospitals play important roles in providing medical and preventive services to low income patients, the Korean government has begun to disseminate newly developed EMR-based information systems to public hospitals as of 2012 to strengthen their service capacity.

New information systems (IS) for public hospitals included a computerized provider order entry (CPOE) system, EMR, admission-discharge-transfer (ADT) system, lab system, a picture archive and communication system (PACS), a customer relationship management (CRM) system, activitybased costing (ABC) system, and data warehouse.

The government has taken a cautious approach to disseminating the new IS by selecting public hospitals which met the following criteria: strong top management support, stable financial status, sufficient IS budget, and sufficient demands for IS (high ratio of outpatients/beds). In addition, the government evaluated the performance of the IS based on the key performance indicators (KPI), such as user satisfaction and reduction in medication errors and personnel costs, before disseminating it to other hospitals.

This study is a part of a government project to evaluate the performance of the new IS in public hospitals that have implemented the system. The performance of the IS was evaluated based on the DeLone and McLean IS Success Model [3], which subdivides success measures into six distinct factors: System Quality, Information Quality, Service Quality, Intention to Use, User Satisfaction, and Net Benefits. Several previous studies have evaluated hospital information systems based on the DeLone and McLean model. Van Der Meijden et al. [4] examined the determinants of success of inpatient clinical information systems according to the DeLone and McLean framework. Park et al. [5] analyzed the performance of the information systems in 38 hospitals by using the DeLone and McLean model, and they found that both System Quality and Information Quality significantly influenced User Satisfaction. Pai and Huang [6] used path analysis for the structured equation model (SEM) to analyze how three quality factors (system, information, and service) influence perceptions of the healthcare information system in terms of usefulness, perceived ease-of-use, and intention to use. They found that three quality factors influenced intention to use through the mediating factors (perceived ease-of-use and perceived usefulness). While Intention to Use is an important success factor, the relationships among IS success should be analyzed by focusing on Net Benefits as outcome factors and using three quality factors as input factors and Intention to Use and User Satisfaction as mediating factors, as suggested by DeLone and McLean for performance evaluation of hospital information systems.

The purpose of this study was to evaluate the performance of the newly developed IS in three public hospitals (Andong, Taegu, and Pohang) that introduced the system in order to determine whether the system was effective in improving user satisfaction and reducing medication errors and personnel costs. Specifically, the KPI satisfaction scores for six IS success factors based on the DeLone and McLean model were compared before and after system introduction. In addition, the relationships among the six IS success factors were analyzed by using path analysis for the SEM in order to identify the important factors influencing three IS success factors (Intention to Use, User Satisfaction, and Net Benefits).

II. Methods

1. Subject

To examine the changes in the KPI satisfaction scores in six IS success factors, two surveys were conducted 171, 225, and 153 users prior to using the system and 140, 232, and 135 users after having experienced the system at three hospitals (Andong, Taegu, and Pohang), as seen in Table 1.

2. Analytic Methods

1) Comparison of key performance indicators before and after system introduction

Based on the DeLone and McLean framework, user satisfaction scores of twelve KPIs for six IS success factors were compared before and after system introduction by using the t-test: System Quality (response time, user friendly screen design, and security and privacy), Information Quality (completeness of medical record, accessibility of clinical information), Service Quality (improved medical service, improved business process), Intention to Use, User Satisfaction (impact of system on organization, improved clinical environment), and Net Benefits (reduction in personnel costs, reduction in medication errors).

2) Relationships among IS success factors assessed by the structural equation model and reliability testing

The relationships among IS success factors were analyzed by SEM, which is a statistical technique for testing and estimating causal relations using a combination of statistical data and qualitative causal assumptions [4]. SEM allows confirmatory modeling which starts out with a hypothesis that is represented in a causal model.

Reliability means the extent to which a measuring instrument contains variable errors, that is, errors that appear inconsistently between observations either during any one measurement procedure or each time a given variable is measured by the same instrument. To test for the reliabilities of factors in the SEM model, composite reliability (CR) was used based on the procedures outlined by Fornell and Larcker [7]. The reliability was acceptable if the CR was 0.70 or higher. The internal consistency reliability of KPIs in six IS success factors were assessed by Cronbach's α. The internal consistency is acceptable if Cronbach's α exceeds 0.6 [8]. SPSS21 and AMOS18 were used for the t-test and path analysis for the SEM.

III. Results

1. Comparison of Key Performance Indicators for IS Success Factors Before and After System Introduction

The satisfaction scores for three KPIs (response time, user friendly design, and security and privacy) of System Quality all increased after the system was implemented at all hospitals (Table 2). Specifically, the scores for security and privacy significantly increased after system introduction at the Andong and Taegu hospitals. The KPI satisfaction scores for two Information Quality factors (completeness of medical record and accessibility of clinical information) showed inconsistent results. While the scores for completeness of medical record significantly increased after system introduction at the Taegu hospital, those of the other two hospitals slightly decreased after system introduction. The KPI satisfaction scores for accessibility of clinical information increased at all hospitals after system introduction. Two KPI satisfaction scores regarding Service Quality (improved service quality and improved quality of business process) improved after system introduction. In particular, the scores for both service quality KPIs significantly increased after system introduction at the Pohang hospital. Scores for Intention to Use increased after system introduction for all hospitals. Scores for two KPIs(impact of system on organization and improved clinical environment) of User Satisfaction increased for the Andong and Pohang hospitals, but the scores for improved clinical environment at the Taegu hospital slightly decreased after system introduction. Finally, the two KPI satisfaction scores for net benefits (reduction in personnel costs and medication errors), which is the system outcome, all increased after system introduction at all hospitals.

2. Relationships among IS Success Factors Assessed by the Structural Equation Model

An SEM analysis was conducted to confirm the hypothesized relationships that were identified by the DeLone and McLean model. As shown in Table 3, Cronbach's α exceeded the acceptable value of 0.6 for internal consistency except for Information Quality (0.57). This indicates that the success factors were reasonably reliable in terms of internal consistency.

The CR of the success factors all exceeded the acceptable value (CR>0.7) for reliability, and the χ²/df value was 1.872, which was less than the acceptable value of 2.0 (Table 4). This suggests that the SEM was a good fit of the internal structure of the model.

In Figure 1, the solid line shows the sets of quality indicators that significantly (at 5% significance level) contributed to User Satisfaction, and the dotted line related to Intention to Use shows a statistically insignificant relationship to net benefits. Specifically, System Quality significantly affected Intention to Use with the path coefficient (or standardized coefficient) β of 0.135 and User Satisfaction with β of 0.180. Information Quality also significantly affected intention to use with β of 0.374 and User Satisfaction with β of 0.135. However, Service Quality did not significantly affect Intention to Use. It only significantly affected User Satisfaction with β of 0.607. Intention to Use did not significantly affect Net Benefits, but User Satisfaction significantly affected net benefits.

The SEM results showed that the three quality measures had a significant linkage to the most widely used success measure, that is, user satisfaction and the intended IS output net benefits. In particular, User Satisfaction had the largest effect on net benefits. However, Service Quality, Intention to Use, and Net Benefits did not have significant linkage.

IV. Discussion

This study evaluated the performance of a newly developed EMR-based information system for three public hospitals by examining the changes in satisfaction scores for twelve KPIs of six success factors based on the DeLone and McLean IS Success Model. Most of the KPI satisfaction scores increased after system introduction. This is in line with the findings of previous studies [4,5]. Kaushal et al. [9] also found that the CPOE system significantly reduced medication errors. However, the satisfaction scores for completeness of medical record and impact on medical services did not increase for some hospitals after system introduction. This is possibly due to the fact that doctors, the primary users of the new system, comprised less than 10% of the total respondents in both surveys and the fact that users did not have enough time to experience the system.

This study also examined the effects of system functions of the newly developed information system (System Quality, Information Quality, and Service Quality) on satisfaction factors (Intention to Use and User Satisfaction) and Net Benefits by using the DeLone and McLean model. We found that all three quality factors had a significant influence on User Satisfaction and that User Satisfaction strongly influenced Net Benefits. The results suggest that hospitals should not only focus on systems and information quality; rather, they should also continuously improve Service Quality to improve User Satisfaction and eventually reach the full potential of IS performance. The results are also consistent with the findings of previous studies. The system quality indicator of user friendly design was a significant factor influencing user satisfaction in the studies conducted by Park et al. [5] and Kim et al. [10]. Pai and Huang [6] also found that when users felt more satisfied with the service quality of healthcare information systems, their satisfaction with usefulness increased.

While System Quality and Information Quality were both significant factors influencing Intention to Use, Service Quality did not have significant linkage with Intention to Use and Net Benefits. This may be due to the fact that the second survey was conducted only one month after system implementation; thus, users did not have enough time to decide whether they intended to use the system or not.

This study has a couple of limitations. First, User Satisfaction, which is the key indicator in IS evaluation, is a measurement of system users' perception. Therefore, there is a possibility of the indicator being subjective. Many studies have mentioned that quantifying measured factors is difficult when evaluating the outcomes of information systems. Furthermore, the second survey was conducted only one month after the system was introduced in three hospitals; therefore, users did not have enough time to experience the system. For the results to be more applicable, the performance of the system should be evaluated again six months after its implementation by using the above results as baseline data.