Journal List > J Korean Soc Med Inform > v.15(3) > 1035535

Park, Chae, Lee, Cho, Kim, and Lee: Evaluation of CDSS for Drug Prescriptions Based on Success Measures

Abstract

Objective

The Clinical Decision Support System (CDSS) for drug prescriptions was developed by integrating the computerized physician order entry (CPOE) system to support doctors and pharmacists in making correct decisions on prescribing drugs in line with the prescription guidelines by the Health Insurance Review Agency (HIRA). The objective of this study was to evaluate the performance of the CDSS with respect to system quality, information quality, and user satisfaction in reducing prescription errors.

Methods

The study was based on survey data from 38 hospitals that were using the CDSS for drug prescriptions. To identify factors influencing the performance of CDSS, multiple linear regression and chi-square analyses were performed.

Results

Regression analysis showed that the variables for system quality and information quality significantly influenced the overall system performance. Specifically, ease of understanding the results and terminology assystem quality measures significantly influenced user satisfaction. Furthermore, based on chi-square analysis, two independent variables (ease of understanding results and decision support functions) were statistically significant with respect to all four dependent variables (information satisfaction, system satisfaction, willingness to recommend to other hospitals, and drug safety).

Conclusion

Based on this study, users should be educated to improve their understanding of the system,and system and information quality should be continuously monitored to improve user satisfaction.

I. Introduction

Most of the information regarding medical care is paper-based and therefore not easily accessible at the point of patient care. Even when the information is present in the form of electronic documents on the web or in clinical information systems, doctors are not realistically given adequate time to search for the information specific to a particular patient sitting in front of them1). This environment is especially prone to an error in ordering drug prescriptions, which indeed is not a rarity in many hospitals2). While most such errors are minor, a small proportion results in an injury or adverse drug event (ADE). While most ADEs are minor -rashes and diarrhea, for example -some are serious, and a few even result in death3).
Amid the information flood that touches every corner of our society, proper management of information and knowledge has become increasingly important. In the medical sector, the amount of information dealt with in the health care field has already well exceeded human cognitive capacity. This suggests that healthcare professionals might find it difficult to memorize, select, and utilize all the needed information at the point of patient care.
The academic advancement in the disciplines of clinical pharmacology, pharmacy, and electronics in the late 20th century has boosted the effort for new drug development, and with it has the significance of drug therapy in medical care increased. However, this rapid increase of new drug development has made it difficult for physicians and pharmacists to obtain complete knowledge about new drugs for patient care. Indeed, in the United States, physicians are seeing patients without extensive knowledge on optimal drug therapy for the following reasons: medical college curriculums have neglected clinical pharmacology, medical training has not been systemic and complete, and there is a gap between medical research and clinical practice4).
The Knowledge Management System (KMS) ensures safe patient care and increases the efficiency of medical service. As part of KMS, the Clinical Decision Support System (CDSS) is a subsystem for providing physicians and health professionals with real-time information based on professional knowledge5). It offers practical help for time-pressed doctors and is increasingly used as Electronic Medical Records (EMR) in hospitals.
Lee et al.6) have developed the CDSS for drug prescriptions and evaluated its effects on medication errors for hospitals. It was designed to assist the physician at the time of ordering and processing by suggesting appropriate doses and frequencies, displaying relevant laboratory data, and screening orders for allergies and drug - drug and drug - laboratory interactions. It also automatically checks whether the prescription order meets the criterion developed by the National Health Insurance Review Agency. In addition, it can assist the pharmacist at the time of ordering processing and consultation by suggesting appropriate doses and relevant medication information. According to the Drugs and Cosmetics Law, pharmacists are required to check for the doses of the prescription order before dispensing and to provide consultation on medication to patients7). Built under the Point of Care (POC) environment, the CDSS was closely integrated with CPOE (computerized physician order entry) in such a way that it provides prescription information at the same CPOE screen on a real-time basis. The CDSS is now being used at 53 hospitals and 3 health centers, most (31) with over 300 beds.
Bates found that medication error rates fell 81% from 142 per 1,000 patient-days in the baseline period to 26.6 per 1,000 patient-days in the final period (p<0.0001) after CPOE was introduced8). In Korea, Kim9) reported that CPOE has affected medication order patterns of physicians and also found reduction in medication errors after CPOE was introduced. Unlike previous studies, we evaluated the performance (or success) of the CDSS based on the Delone and McLean framework10). Delone and McLean subdivided success measures of information systems into six distinct categories: (1) system quality, (2) information quality, (3) user satisfaction, (4) usage, (5) individual impact, and (6) organizational impact. Meijden et al.11) examined the determinants of success of inpatient clinical information systems according to the Delone and McLean framework. Shin et al.12) also used the Delone and McLean framework to analyze the factors influencing the success of ERP (Enterprise Resource Planning) system for hospitals.
The purpose of this paper was to analyze the effects of two quality measures (system and information) on CDSS performance, including user satisfaction and improvements in drug safety. Based on the results, we recommended the areas for further improvement of the system.

II. Materials and Methods

1. Subject

A questionnaire was designed by structuring primary evaluation areas first and then more detailed sub-areas in each primary area. The survey was conducted by mail or by visit for 6 weeks from April 20 to May 29, 2009, for the pharmacists from 38 hospitals using the CDSS. A total of 84 questionnaires were returned from 22 users (response rate of 58%). Of them, 77 were used for the analysis, excluding 7 that were found unusable for statistical analysis.

2. Model and statistical analysis

Based on the Delone and McLean framework, we evaluated the success of the CDSS by using two quality measures (system quality and information quality) as independent variables; and three performance measures (user satisfaction and organizational impact) as dependent variables for the multiple regression analysis (Fig. 1). In this study, usage and individual impact measures suggested by Delone and McLean were not included in the analysis because we did not consider them critical measures of the CDSS. Three measures were used for evaluating system quality: ease of system use, ease of understanding results, and ease of understanding terminology. In addition, five measures were used for evaluating information quality: information accuracy, timeliness, credibility, up-to-datedness, and decision-supporting.
In addition, chi-square analysis with cross-tabulation was performed to analyze the specific associations among two quality measures and three measures of user satisfaction (information satisfaction, system satisfaction, and willingness to recommend to others). We also analyzed the associations between two quality measures and improvements in drug safety as a performance measure of organizational impact. The SAS 9.1 package was used for statistical analysis of the questionnaire data.

III. Results

1. Characteristics of the subjects

Most of survey respondents were female, working-level pharmacists with college degrees between 30 and 40 years old (Table 1). Most respondents (n=61; 79.2%) were working for either teaching or general hospitals.

2. Analysis of the quality measures influencing system performance

(1) The effects of system quality and information quality on system performance

We examined the effects of two quality measures on the overall system performance by combining user satisfaction and improvement in drug safety. The results of the multiple regression analysis showed that both system quality and information quality significantly influenced the system performance. System quality was slightly more significant than information quality measure (Table 2). The overall R-square was 0.244.

(2) The effects of quality measures on user satisfaction

Of the three system quality measures, ease of understanding results and ease of understanding terminology significantly influenced user satisfaction, but ease of system use did not. The R-square of this model was 0.391 (Table 3).
Of five information quality measures, however, only decision supporting function significantly influenced user satisfaction, with an R-square of 0.455 (Table 4). This implies that decision supporting is the most important function for CDSS, whereas timeliness or other factors may be more important functions for CPOE.

3. Analysis of association between quality measures and user satisfaction

(1) Association between quality measures and information satisfaction

While all system quality measures were significantly associated with information satisfaction, ease of terminology understandability was not positively (or consistently) associated with information satisfaction (Table 5). That is, of the highly satisfied respondents on information, 66.7% of the respondents did not understand terminology; 28.2% of them slightly understood terminology; and 80% of them highly understood terminology.
All information quality measures were significantly associated with information satisfaction and highly satisfied respondents on information also positively responded to information quality. However, only the decision supporting function had higher percentage of positive respondents (55.8%) than in-between respondents (41.6%).

(2) Association between quality measures and overall satisfaction

All quality measures except ease of the system use were significantly associated with the overall satisfaction (Table 6). Specifically, ease of understanding results received a positive response from 45 respondents (58.4%), of whom 66.7% also gave a positive response to the overall satisfaction with the system. Of the information quality measures, decision making support functions obtained a positive response from 43 (55.8%), of whom 65.1% also gave a positive response to the overall satisfaction. This analysis showed that ease of understanding results for system quality and decision supporting function for information quality were the two biggest contributors to the increase in overall satisfaction.

(3) Association between quality measures and willingness to recommend to others

All quality measures except information accuracy were significantly associated with the willingness to recommend it to others (Table 7). Specifically, of the system quality measures, ease of understanding results obtained a positive response from 45 respondents (58.4%), of whom 97.8% also gave a positive response (average or higher) to the willingness to recommend to others. Of the information quality measures, decision supporting function obtained a positive response from 43 respondents (55.8%), of whom 97.7% also gave a positive response (average or higher) to the willingness to recommend to others (Table 7). This analysis showed that ease of understanding results and the decision supporting function were also the two biggest contributors to willingness to recommend to others.

4. Analysis of association between quality measures and improvement in drug safety

Only ease of system use in system quality measures and two information quality measures (up-to-datedness and decision supporting) were significantly associated with improvements in drug safety (Table 8). Specifically, information accuracy showed inconsistent association with the improvement in drug safety. Of the system quality measures, ease of understanding results showed the highest positive responses (45 respondents, 58.4%), of whom 93.3% also gave a positive response (average or higher) to the improvement in drug safety. Of the information quality measures, decision supporting function gained the highest positive responses (43 respondents, 55.8%), of whom 95.3% also gave a positive response (average or higher) to the improvement in drug safety. Thus, ease of understanding results and decision supporting function were the two biggest contributors to improvements in drug safety.

IV. Discussion

This study evaluated CDSS performance for drug prescriptions according to the Delone and McLean framework10). We found that system quality and information quality significantly affected system performance. User satisfaction was significantly influenced by the ease of understanding results and terminology as system quality measures, and decision supporting function as an information quality measure. We also found that ease of understanding results and decision supporting function were the biggest contributors to the increase in every aspect of system performance. In other words, the two factors were the key determinants of CDSS success.
Our study showed that the dimensions of success defined by Delone and McLean for information management systems are applicable to the CDSS for drug prescriptions, similar to a study on inpatient care information systems by Meijden et al.11). The results of this study also support the findings by Shin et al.12), who found that both system quality and information quality measures significantly influenced ERP performance at the study hospital.
We therefore suggest the following measures to improve CDSS performance. First, systematic user training is required to help them learn how to use the CDSS and understand the results and terminology, system functions, or error messages, to achieve the success measures suggested by Delone and McLean. Furthermore, the system quality and information quality measures that significantly affect the CDSS need to be thoroughly revisited and corrected for optimal operation of the system. Especially, ease of understanding results and decision supporting function should be further strengthened to improve user satisfaction.
There are several limitations in this study. First, this study only surveyed hospitals using a certain type of CDSS, and may not be generalizable to other systems. Second, while pharmacists are important users of the CDSS, this study did not include physicians in the survey, and they may have different opinions on the CDSS system. Future studies should incorporate physician feedback on the system, particularly focusing on decision supporting functions.

Figures and Tables

Figure 1
Research framework for the evaluation of the CDSS.
jksmi-15-293-g001
Table 1
General characteristics of the respondents (Unit: persons, %)
jksmi-15-293-i001
Table 2
The effects of quality measures on system performance
jksmi-15-293-i002

R-Square=0.244

p value<.0001

Table 3
The effect of system quality measures on user satisfaction
jksmi-15-293-i003

R-Square=0.391

p value<.0001

Table 4
The effect of the information quality measures on user Satisfaction
jksmi-15-293-i004

R-Square=0.455

p value<.0001

Table 5
The association of quality measures and information satisfaction (Unit: persons, %)
jksmi-15-293-i005
Table 6
The association between quality measures and overall user satisfaction (Unit: persons, %)
jksmi-15-293-i006
Table 7
The association between quality measures and willingness to recommend to others (Unit: persons, %)
jksmi-15-293-i007
Table 8
The association between quality measures and improvement in drug safety (Unit: persons, %)
jksmi-15-293-i008

References

1. Musen M, Shahar Y, Shortliffe E. Shortliffe EH, Cimino JJ, editors. Biomedical informatics. 2006. 3th ed. New York: Springer.
2. Barker KN, Allan EL. Research on drug-use-system errors. Am J Health Syst Pharm. 1995. 52(4):400–403.
crossref
3. Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape LL. Relationship between medication errors and adverse drug events. J Gen Intern Med. 1995. 10(4):199–205.
crossref
4. Park BJ. Drug therapy and drug utilization review. J Korea Med Asso. 1994. 37(2):181–188.
5. Classen DC. Clinical decision support system to improve clinical practice and quality of care. J Am Med Asso. 1998. 280(15):1360–1361.
crossref
6. Lee YT, Bae MY, Park JH, Choi CK, Bae SB, Chae YM. Evaluation of the effects of the CDSS for drug prescription for hospitals. J Korea Soc Health Infor Stat. 2007. 32(2):89–98.
7. Park BJ. Introduction of drug utilization review. J Korea Med Asso. 2004. 47(2):108–111.
8. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Luf NM, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999. 6(4):313–321.
crossref
9. Kim CS. Study of drug utilization pattern after introduction of computerized physician order entry system. 2004. Sookmyung University Graduate School of Pharmacy;Masters dissertation.
10. Delone WH, McLean ER. Information system success: the quest for the dependent variable. Inf Sys Res. 1992. 3(1):55–69.
11. Meijden MJV, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc. 2003. 10(3):235–243.
crossref
12. Shin EJ, Chae YM, Ho SH, Kim YU. Performance of ERP system in a hospital. J Korea Soc Med Infor. 2007. 13(2):77–82.
TOOLS
Similar articles