A drug–drug interaction (DDI) alert is a type of medication-related clinical decision support function in a computerized provider order entry (CPOE) system. These alerts provide the promise of substantially reducing the number of adverse drug events (ADEs) by introducing automation at the time of ordering and by rapidly supplying usable prescribing algorithms (1). However, implementations of DDI alerts do not always provide the anticipated benefits. They can be very difficult to optimize, and alert fatigue can be a major problem if too many false-positive warnings are delivered (2).

Since December 2010, Korean hospitals have been required to provide DDI alerts to physicians using the prospective drug utilization review (DUR) system run by the Health Insurance Review Agency (HIRA) (345). This DUR system aims to provide quality assurance and ensure the provision of appropriate drug therapies, and it was designed to prevent potential ADEs arising from medication errors and inappropriate drug use (35). The national DDI rule set was initiated and included 162 DDI combinations and up to 706 contraindicated co-prescription pairs as of August 2015.

While DDI alerts can reduce the number of DDI-related ADEs, alert fatigue induced by large numbers of DDI alerts can also lead to physicians ignoring clinically significant DDI alerts (2678). A recent review of empirical analyses of CPOEs with the national DDI rules found that physicians in general hospitals overrode a substantial fraction of automated warnings, with one study finding that physicians continuing with prescription orders in 72.2% of DDI cases (9). One study of DDI alert logs revealed a high override rate of 72.8% at a tertiary hospital, even the authors just considered a new alert regarding a DDI during 18 months of observation period (10). Our previous examination of DDI alert logs revealed a high override rate of 83% at a tertiary hospital (8). According to a report which analyzed HIRA DUR data, the override rates for DDI alerts were 76.6% in 2012 and 78% in 2013 (11). Such high override rates have prompted serious concerns that physicians may be overriding or ignoring clinically important warnings, which potentially has major implications both for safety and physician liability. The alerts may be unnecessarily interrupting physicians, which could threaten the provision of a consistent work process and cause inefficient use of time and other resources (12).

In an attempt to address this problem, the present study investigated the national DDI rules using three approaches. The first approach involved comparing the Korean DDI rules with a list of the most dangerous DDIs. These lists, known as high-priority DDI rules, were identified by a sponsor of the Office of the National Coordinator for Health Information Technology in the USA (1314). It is a set of 15 high-severity, clinically significant DDIs were identified for which it was advised that warnings should be generated in all electronic health record systems. The second approach involved investigating the DDI alert log data of a tertiary hospital to identify the rates at which high-priority DDI rules are alerted and overridden in practice. We also compared the results with those obtained at three hospitals in Belgium, the UK, and the USA. The third approach involved examining medication orders from two other hospitals in order to determine whether unmatched high-priority DDI rules appear in prescriptions. Based on the obtained results, we describe several areas that need further investigation in order to achieve the successful application of DDI alerts.

We compared the current Korean national DDI rules with the lists of high-priority DDI rules found in previous studies (13). To identify the differences, we use a comparison framework (Fig. 1). A retrospective observational design was used to analyze the alert system logs. Fig. 2 shows the study procedure, materials, measurements, and participating hospitals. We investigated data from three Korean hospitals: data from one hospital were used for analyzing alert log data, and the data from the other two were used for analyzing prescription data. The intercountry analyses compared the performance of national DDI rules in South Korea with those of hospitals in three other countries. In this study we included a part of results of another study involving three of the authors (I.C., JH.L. and D.W.B.) (8).

We found that the national rules coverage rate for high priority DDI rule was 3.0%, which was low given that there were 706 contraindicated rules. These matched high-priority DDI rules that triggered 0.3% of the alerts that the physicians had received, and these showed a lower override rate than the overall override rate. The proportion of high-priority DDI rules among the Korean national rules was lower than for those of the UK and USA. We also found noticeable numbers of unmatched high-priority DDI pairs in prescriptions in two inpatient settings, which may be associated with medication safety risks.

The national rules included 80% of the high-priority DDI rules at the class level, but omitted most of the possible high-priority DDI pairs. These findings imply that the national rules include sufficient drug classes for high-priority DDI rules but are deficient at the drug level, even though the total number of rules exceeded 700. We can think of two reasons for this situation. The first is that unmatched high-priority DDI pairs are not prescribed by physicians in Korea, perhaps due to their good education or training. However, our findings of noticeable numbers of unmatched high-priority DDI pairs in prescriptions for inpatients make this hypothesis unlikely. In addition, the small degree of overlap of the unmatched rules between the two hospitals suggests that investigating different hospitals could reveal different unmatched high-priority DDI pairs. Therefore, this might not due simply to the availability in a drug formulary at each site. The second possibility is that the expert committee of HIRA did not consider the unmatched high-priority DDI rules. If that is the case, the unmatched rules should be considered first for inclusion in the national rules. The level of risks of the moderate-priority DDI rules then needs to be considered. We doubt that DDI rules with a moderate level of risk are appropriate in all circumstances, such as “always contraindicated DDIs,” but such a list would be of limited usefulness because it could not adequately reflect the clinical complexity of many treatment decisions, especially in tertiary hospitals. The risk of DDIs often varies from patient to patient, potentially being influenced by factors such as age, sex, kidney function, liver function, genetic makeup, and diseases, as well as the dose and route of administration of the medication (151617).

Our previous investigation of the appropriateness of DDI alert overrides by physicians based on a review of patient EMRs revealed that 68.2% of the DDI alert overrides were considered appropriate (2). This relative high rate of appropriateness implies that simple checking of prescription drug lists could have limitations, because this would not consider the clinical context and patient-specific information. Adapting rules based on patient-specific information could improve the clinical sensitivity of DDI rules and alerts.

Considering the matched high-priority DDI rules, only 7 rules triggered 66 alerts, which represented only 0.3% (66/21,889) of all alerts, and physicians responded to many of these. The override rate of 51.5% is lower than the overall override rate of 79.6%, but none of these alerts should actually have been overridden. This result implies that physicians rarely prescribe drugs involved in high-priority DDIs together, and when they receive alerts related to high-priority DDI rules they are more willing to accept them. Therefore, including more high-priority DDI rules in the national rules might improve the override rate and relieve alert fatigue, as well as reduce interruptions to the work flow of physicians and time-wasting. In addition, organizations might want to consider “hard stops” so that for the very most severe of these, the drugs cannot be ordered together.

The national DDI rules were developed by a process in which the DDIs to be avoided were determined, and involved experts representing various professional organizations (1). However, it should be remembered that there is considerable debate about the scientific evidence for the risk that many DDIs pose to patients (13). DDI information can be obtained from a various sources, but the data are often inconsistent or unreliable. One study of 458 drug pairs found that proprietary databases rated 7.4% of interactions (34/458) as severe while clinicians assessed 6.6% (30/458) as severe; this looks promising, but unfortunately only 3 of the interactions were considered severe according to both the databases and clinicians (18). Another study found that a panel of experts initially agreed on the clinical significance of particular DDIs approximately 50% of the time (19). The comparison of high-priority DDI rules using log data obtained in different countries revealed that even though the coverage of high-priority DDI rules is lower in Korea than in the other investigated countries, the override rates in Korea was similar with that of USA, and were much lower than those in UK and Belgium. This finding implies that Korean physicians responded more likely to accept the alerts of high-priority rules.

We also noticed that the two categories of QT-prolonging agent pairs and the HMG Co-A reductase inhibitors–CYP3A4 inhibitors showed the greatest differences among countries. The Belgium hospital had the largest number of rules for those pairs, but also had a low alert frequency. In contrast, the USA hospital had smaller numbers of such pairs but the largest numbers of alerts with a very low override rate compared to Belgium. The UK hospital had QT-prolonging agents rule, but these inactivated in their system, and only five rules were present for HMG Co-A reductase inhibitors–CYP3A4 inhibitors, with no cases of alerts. The Korean national rules contained small numbers of those pairs with much smaller alert frequencies, but higher override rates than in the USA. We did not know whether or not the high override rate was due to the Korean national rules having smaller rule sets for these class-pairs. However, based on the frequency analysis of drug pairs that were not matched with high-priority DDI rules in prescriptions, it is likely that appropriate rules were missing.

Our examination of unmatched high-priority DDI pairs in prescriptions showed that significant numbers of drug combinations were prescribed. Most of the combinations were QT-prolonging agent pairs, and they constituted 0.5%–0.6% of the total prescriptions at two of the hospitals. For the categories of rifampin–ritonavir and HMG Co-A reductase inhibitors–protease inhibitors, only small numbers of prescriptions were observed in one hospital. Although these are small proportions, they are potentially associated with high DDI risks and their consequences could be serious or even fatal; they therefore need to be considered for incorporation in the national rules.

In summary, the Korean national rules are shortcoming in coverage of contraindications of high-priority rules, which could be responsible for problems of a high override rate and alert fatigue associated with physician liability. When we assessed the rate of overrides by physicians, about 15% of them (80/529) received more than 20 DDI alerts during the 4-month analysis period and overrode all of them; that is, an override rate of 100%. The reason we used the frequency of 20 DDI alerts during the period, was to exclude the miss classification of physicians with small number of alerts having relatively high override opportunity. The 100% override was both unexpected and undesirable, which demonstrates the need to consider several aspects such as the provision of too many unnecessary interruptions and an inadequate consideration of liability. Our findings also identified several possible ways for minimizing the alert fatigue associated with the national rules in tertiary hospitals, including strategies to increase their sensitivity and relevancy. These suggestions involve both government and local hospitals. First, it would actually be unreasonable to expect one set of rules to fit all hospitals due to the diversity of patients with vastly different types of illnesses and levels of severity. An approach based on the DDI priority level would be easier than one based on patient severity, because of a lack of measurements, tools, and interrater reliability. For example, high-priority DDI rules could be considered for all hospitals with a hard stop or interruptive, while moderate-priority DDI rules would be presented as being optional or informative in tertiary hospitals (4).

Second, DDI warnings should provide concise but comprehensive information that will enable physicians to make more knowledgeable and effective prescribing decisions (4). The current simple lists of DDI combinations that are typically used can be misleading, as they do not address clinical complexity or the highly individualized nature of many treatment choices. Such information could include alternative drugs, ways to adjust patient monitoring, and general background information such as evidence of DDIs, the clinical relevance of the possible adverse DDI consequences, specific risk factors, and the incident rate of adverse reactions.

Third, to improve the rule sensitivity, each hospital could utilize patient data stored in their own EMR systems for determining when DDI warnings will be issued. Both the risk of an adverse drug reaction due to a DDI and the quality of the evidence concerning that risk may depend on multiple patient-specific risk factors, some or even all of which may be recorded in the EMR of a patient. The reliability of DDI alerts will then depend on the accuracy of the data in EMRs, which may be compromised by input errors, omissions, and incomplete entries. One research reported that adding clinical laboratory values, concomitant medication, patient demographics, and administration time to basic DDI checking algorithm results in a 55% reduction of the number of alerts and a 45% reduction in time spent on DDI checking, yielding a return on investment of almost 10 years (20). Also, before being deployed, the rules should be piloted in several representative settings to determine whether they are effective in improving prescribing decisions and are not overly disruptive to physicians. Additional or revised rules could be incorporated in the prescribing process through a similar process. These processes could be undertaken by the HIRA.

This study was subject to several limitations. First, practice log and prescription data obtained from three tertiary teaching hospitals were used, and so the results might not be generalizable to other tertiary general hospitals. However, the national rules are mandatory and they are used nationwide in all primary, secondary, and tertiary hospitals. Moreover, our findings were consistent with those in other literature and relevant reports. We also compared our findings in the present study with findings that we obtained in other countries in previous studies. Second, we did not evaluate the appropriateness of the alerts that were overridden and could not follow up the patient outcomes in the overridden cases; further research is therefore needed in order to optimize the national DDI rules.

The current national DDI rules of HIRA are not complete in terms of their severity coverage. They also have considerable shortcomings in clinical efficiency in tertiary settings, which indicates the need for improved systematic approaches (21). To improve safety and reduce alert fatigue, the national rules should be improved by setting up a process of monitoring and refinement of the list, establishing tests and validation processes before releasing them, empowering physicians by providing them with concise and comprehensive information, and using a sophisticated design that is integrated with the patient data available in EMR systems.