Journal List > J Korean Med Sci > v.31(Suppl 1) > 1023346

Jeong, Baek, Jung, Do, Yong, Kim, Choi, Lee, Choi, Jo, and Choi: Imaging Guidelines for Enhancing Justifications for Radiologic Studies

Abstract

Justification in the field of radiology refers to the appropriate use of radiologic imaging modalities, and may be achieved by establishing clinical imaging guidelines (CIGs). Recently, CIGs have been shown to be useful in selecting the proper medical imaging modality, resulting in the reduction of inappropriate radiologic examinations, thereby enhancing justifications. However, the development of CIGs is both time-consuming and difficult as the methodology of evidence-based medicine should be adhered to. Thus, although the radiologic societies in developed countries such as the United Kingdom and USA are already developing and implementing CIGs in their clinical practices, CIGs are not yet readily available in many other countries owing to differences in medical circumstances and resources. In this review, we assess the role and limitations of CIGs by examining the current status of CIGs in developed countries, and also describe the specific efforts made to establish CIGs in Korea.

Graphical Abstract

jkms-31-S38-abf001

INTRODUCTION

Justification and optimization are the two fundamental principles of medical radiation protection for patients who may need to undergo a radiologic examination. Justification involves the clear delineation of whether a radiologic examination may indeed be reasonable or necessary in any given situation, and is used in tandem with optimization which is the action of what constitutes “reasonable” use in clinical practice. Although there has been relatively much progress in the optimization aspect of radiology, justification has been less emphasized until now (1). With the rapid development of radiology in modern medicine, we have seen an explosive expansion of radiologic examinations, and several related problems have come to the fore (23). Among them, inappropriate imaging has been the most representative issue. There are various causes of inappropriate imaging: when visiting a hospital, patients often have an expectation that something such as CT or sonography should be done by the practitioners, while some physicians fear potential liability exposure owing to a delayed diagnosis, imaging is scheduled. Also the self-referrals, in which institutions refer an imaging study to themselves, have become an emerging cause of inappropriate imaging. Clearly, the conflict of interest presented by physician ownership of imaging equipment has been interlinked with low medical cost of Korea: too many radiographic machines such as ultrasound, CT, MRI and mammographic machines are installed, and it has recently become an important social issue in Korea. In addition, the increased number of radiologic examinations increases medical costs and also increases radiation exposure, which is well known to pose a risk of health problems including cancer development (4). Therefore, to lessen the risk of radiation exposure, justification of radiologic examinations has become an important prerequisite, wherein the benefits should substantially outweigh any risks that may be incurred (5). The benefits and risks of radiation exposure are generally well agreed upon within the medical profession including the radiologic society, and should also be informed to patients.
Since 2002, the International Atomic Energy Agency (IAEA) has made great efforts to disseminate the importance of the justification of medical radiation exposure. Through two international consultancies, IAEA and World Health Organization (WHO), three key issues for the implementation of justification have been suggested: Awareness, Appropriateness, and Audit. The “3As” were also endorsed as action plans for justification by the steering panel of the IAEA (Bonn Call-for-Action). For justification in clinical practice, evidence-based clinical guidelines for the referral of imaging studies are a useful method as they can help clinicians refer to appropriate imaging and interventional procedures in each situation, helping to obtain the best diagnostic information with a minimum of radiation. ‘Evidence-based’ means a systematic approach to assimilate answers about key clinical questions through in-depth review of all of the available literature (67). It is different from the previous form of clinical practice, or so called ‘eminence-based’ practice, which is based on an individual’s experience, which can be biased by overemphasizing more recent events, with a small sample size, and a lack of objectivity. Clinical decision making, including what imaging modality can be used for any specific situation, should be based on the best current evidence with consideration of radiation exposure. ‘Evidence-based’ guidelines for the appropriate use of imaging studies can lead to increased communication of awareness to the patient, facilitate decision support, and avoid unnecessary practices. Furthermore, deployment of clinical imaging guidelines (CIGs) can contribute to successful protection from unnecessary medical radiation exposure as well as effective distribution of limited medical resources. Several countries, especially in the British Commonwealth and the United States, have developed a form of CIGs since the 1990s, and several reports have shown that the guidelines, clinical audits and feedbacks have decreased the overall referrals of radiologic imaging while preserving the quality of referrals (8910). This review article will introduce some of the CIGs used for the justification of radiologic studies, and explain the current status of the development and implementation of CIGs in these countries and their existing limitations. Furthermore, we will present the specific efforts for justifications in radiology in Korea.

CLINICAL IMAGING GUIDELINES: A PRACTICAL METHOD FOR ENHANCING JUSTIFICATION

Background of clinical imaging guidelines: evidence-based radiology

With much clinical research performed and their results shared on-line, clinical practice based on evidence obtained through research, or evidence-based medicine (EBM), has recently emerged in healthcare. EBM is the use of mathematical estimates of risk derived from high-quality research on population samples to inform the benefit and harm of clinical decision making in the diagnosis, investigation or management of individual patients. In the radiologic field, CIGs based on evidence have been developed and brought justification to the decision making by physicians who refer the radiologic examinations. Evidence-based radiology (EBR), on the other hand, a version of EBM, has different points compared with conventional EBM (7). First, radiologic studies are usually different from other medical studies as most are focused on the diagnostic performance of the imaging modality, rather than the therapeutic outcome of the patient (1112). Second, the rapid technological evolution of the radiologic field has brought about not only the need for education of its basic theory and practical applications of new tools, but also to perform studies on the technical and diagnostic performance of the new modalities. Thus, it has not been easy to progress to the next step, i.e., to evaluate the patients’ outcome. Third, the reproducibility issue of imaging modalities (intraobserver, interobserver and inter-study variability) is important, so there are dedicated study designs and statistical methods (e.g., Kappa statistics) for radiologic studies. Last, there is the need to avoid unnecessary exposure to ionizing radiation, according to the as low as reasonably achievable (ALARA) principle and government regulations. Although these practical difficulties may impede EBR, there have been many efforts to achieve high quality evidence development in radiology including systematic reviews and clinical trials in recent years.
The development of CIGs is an important part of EBR. There have been many concerns of inappropriate imaging in clinical care, especially regarding the efficacy, cost and radiation exposure of imaging modalities. To that end, several different organizations such as the American College of Radiology (ACR), the United Kingdom Royal College of Radiologists (UK RCR), the European Society of Radiology (ESR), and the Korean Society of Radiology (KSR) have pursued the development of CIGs. Representative guidelines include iRefer by the UK RCR and the Appropriateness Criteria® suggested by the ACR, and many national radiologic societies have since adopted or adapted these guidelines for application in their countries.

Clinical imaging guidelines in the United Kingdom: iRefer

iRefer, a referral guideline by the RCR in the United Kingdom, has been developed and updated since 1989 under the banner of ‘Making the best use of clinical radiology services’, and has been available in two formats, on paper and online since its 7th edition (13) (Table 1). It has helped the referring physicians in primary and secondary medical care to select the most reasonable imaging modality for diagnosis. There have been many investigations on the effectiveness of this guideline, and it has been shown that the overall number of referrals and effective radiation doses has declined with the adoption of iRefer (8910). iRefer provides 306 evidence-based guidelines in 12 systemic sections including cancer, interventional radiology, and trauma. The development process of the guideline has been accredited by the National Health Service (NHS) Evidence, and managed by the National Institute for Health and Care Excellence (NICE). To gather the evidence, a search of the English language literature was made through Medline, EMbase, the Cochrane Library, NICE, the Scottish Intercollegiate Guidelines Network (SIGN) and ACR Appropriateness Criteria®. The level of evidence was divided into A, B, and C, based on the Oxford Centre of Evidence-based Medicine (14). The recommendations of this guideline can be divided into four categories: indicated, specialized investigation, indicated only in specific circumstance, and not indicated. Thereafter, agreement of the recommendations was achieved through the Delphi process. The format of the guidelines consisted of clinical/diagnostic problems, the sort of investigation performed, radiation dose, recommendation grades, and specific comments (13). The typical dose of irradiation during investigation was indicated by symbols with five grades; no radiation (none), lower than 1 millisievert (mSv), 1-5 mSv, 5.1-10mSv, and higher than 10 mSv based on the dose information provided by the Public Health England (PHE). Unfortunately, iRefer does not allow access to parts of the core information such as evidence to other developers although the guidelines were developed in an evidence-based manner. Nevertheless, it is used by more than 20 other countries through adoption and translation of the guidelines.
Table 1

Comparison of representative clinical imaging guidelines between the UK and USA

jkms-31-S38-i001
Items iRefer (UK) ACR Appropriateness Criteria (USA)
First version Since 1989 Since 1995
Number of guidelines 12 categories, 306 guidelines Diagnostic part: 11 system, 168 guidelines (excluding radiologic oncology)
Grading of evidence Based on Oxford center for evidence based medicine: A to C Based on the method developed by ACR: category 1 to 4
Grading of recommendation 4-grade system 9-degree systems
Indicated 1-3 usually not appropriate
Specialized investigation 4-6 may be appropriate
Indicated only in specific circumstance 7-9 usually appropriate
Not indicated
Radiation exposure 5 grades (0 to 4+) 6 grades (0 to 5+)
Accessibility Less accessible (access after purchase) More accessible (open access via web for free)
Booklet, online, and application for mobile Online version
Evidence tables are not opened Evidence tables are opened

Clinical imaging guidelines in the United States: ACR appropriateness criteria®

The first volume of the ACR Appropriateness Criteria® (AC) was published in 1995 (15) (Table 1). It was developed as an acceptable medical practice guideline used by the Agency for Healthcare Research and Quality (AHRQ) as designed by the Institute of Medicine, in order to assist in the decision making of appropriate radiologic imaging in conjunction with radiation oncology by the American College of Radiology (16). Compared to iRefer, its advantages are its accessibility of the recommendations and evidence tables on-line and its support of a digital decision support system, also referred to as ACR Select. In the 2000s, the ACR extended the coverage of the guidelines and made an effort to disseminate it in general practice in collaboration with the Radiological Society of North America (RSNA). For the development and extension of the guidelines for radiologic imaging, the ACR committee on AC was organized under the ACR’s Commission on Quality and Safety, and separated into two committees, the Committee on Diagnostic Imaging (DI)/Interventional Radiology (IR) AC. These committees consist of diagnostic panels organized according to the body system, such as the breast, cardiac, gastrointestinal, musculoskeletal, neuroradiology, thoracic, urologic, vascular, pediatric and women’s imaging. The subcommittees on radiation exposure and methodology provided the relative radiation levels for the procedures and methodological oversight to all of the panels.
The ACR AC addresses 208 clinical conditions with more than 1,000 variants, and new topics have been added to reflect changes in technology and clinical practice over the years. The guidelines are based on a systematic review of evidence as demonstrated by literature search, evidence table development, and topic development documents, and has been updated by the panels every 3 years or sooner. For rating of the guidelines, a modified Delphi method was used following a series of surveys conducted to elicit experts’ interpretation of the evidence regarding the risks or harms of doing the procedure. The ACR AC is represented in an ordinal scale that uses integers from 1 to 9 grouped into three categories: 1 to 3, “usually not appropriate” where the harms of doing the procedure outweigh the benefits; 4 to 6, “may be appropriate” where the risks and benefits are equivocal or unclear; and 7 to 9, “usually appropriate” where the benefits of doing a procedure outweigh the harms or risks.
In 2012, ACR Select, a web-service version of ACR AC, was introduced by a collaboration of ACR and the National Decision Support Company (NDSC) (17). This clinical decision supporting system was also integrated with electronic health or medical records (EHR/EMR) and computerized provider order entry (CPOE) systems, so healthcare organizations can effortlessly utilize ACR AC ensuring that the physician can easily select appropriate imaging modalities on-line. This system not only helps the primary physician use ACR AC but also helps the developer of the guidelines to obtain information regarding user experience such as application of new imaging procedures and feedback. Indeed, one recent report showed that integrating clinical decision support to inpatient CPOE improved the appropriateness of advanced imaging requests and was more helpful for primary physicians than specialists (18).

Clinical imaging guidelines in European Society of Radiology: EuroSafe Imaging

The European Society of Radiology (ESR) has also made efforts to reduce medical radiation exposure and to promote more appropriate use of medical imaging with a campaign entitled EuroSafe Imaging (19). It consists of a monthly survey on CT diagnostic reference levels (DRL) for different indications (“Is your Imaging EuroSafe?” project), European Medical ALARA Network (EMAN) focused on optimization, MEDicalRAdiation Protection Education and Training (MEDRAPET), referral guideline project addressing the development and implementation of CIGs, and European Diagnostic Reference Levels for Paediatric Imaging (PiDRL). For the Referral guideline project, ESR and consortium partners including the UK RCR, Société Française de Radiologie (SFR), Cardiovascular and Interventional Radiology Society of Europe (CIRSE), and European Society of Paediatric Radiology (ESPR) held a workshop to obtain referral guidelines for imaging in 2012.
In Europe, Council Directive 97/43/Euratom on the health protection of individuals against the potential risks of medical radiation in relation to medical use was established in 1997. Thereafter, the European Commission issued a referral guideline entitled the “Referral Guideline for Imaging” in 2000, and revised it in 2008. This guideline was adopted from the RCR, and evidence, recommendation and radiation exposure levels were based on the same system as iRefer. Subsequently, there have been several efforts for the implementation of Council Directive 97/43/Euratom requirements concerning referral criteria for medical imaging in the European Union. In 2012, a European Commission Referral Guideline Workshop was held, and participants from 30 EU countries and international organizations such as the IAEA, WHO, and European Association of Nuclear Medicine (EANM) presented about and discussed the current status of referral guidelines in Europe, on how to drive the effective use of the guidelines, and the challenges in the production of evidence-based guidelines, as well as difficulties in their implementations (20).

Limitations of clinical imaging guidelines

The IAEA has annually held the Technical Meetings for Justification to improve the international collaborations for efficient worldwide implementation of CIGs since 2011. In the meeting, problems regarding the development and implementation of CIGs were addressed (Fig. 1). First, de novo development of evidence-based guidelines was reported to be time-consuming and difficult, therefore only a few professional organizations in a few developed countries have been able to make the long-term financial, personnel, scientific, and political commitment to develop guidelines appropriate for their medical situations. The type of radiologic services was also shown to affect the development and deployment of the guidelines. Each country also has different health care systems and clinical practices compared to other countries. As an example, in Korea, clinical physicians generally decide and prescribe the imaging studies themselves rather than referring to radiologists. Subsequently, the roles of radiologists remain passive in the context of being gatekeepers for medical radiation protection. General physicians who carry out clinical decisions on imaging studies do not usually perceive the necessity of CIGs and the need to deploy them in practice in Korea. Furthermore, there are practical differences in the implementation of the guidelines between developed and developing countries. In developed countries, it is easier to access alternative imaging methods, such as MRI or ultrasonography when computed tomography is not proper in certain situations. Finally, there is an issue regarding the diversity of stakeholders, which is very important in the implementation of CIGs. To activate CIGs, all stakeholders including radiologists, referral clinicians, government personnel, payers, and patients should agree on the necessity of CIGs, but there are complicated relationships among the stakeholders, which may be different between countries.
Fig. 1
Limitations and solutions for the implementation of clinical imaging guidelines.
jkms-31-S38-g001
To overcome these limitations, the technical meeting in IAEA suggested a different method for the development of CIGs, adaptation. Adaptation is the modification and introduction of pre-developed CIGs for use in other countries and it may be the best solution when all issues derived from de novo development of CIGs are considered. Simple translation or adoption of developed CIGs may be much easier, but these methods are not flexible enough to reflect the different local situations including medical service systems and availability of imaging modalities. Although it may not be easy to overcome the diversity of medical circumstances, nevertheless, technologic developments such as the clinical decision supporting system (CDSS) using information technology (IT) systems in hospitals can help distribute and familiarize CIGs to users including radiologists, radiographers and physicians. Governance support of CIG is another key to implementation of these guidelines (21). Ideally, health authorities must help to implement CIGs in clinical practice as they can take actions necessary to ensure that guidelines are used and also audit the use of CIGs. They can also arrange a reimbursement system and establish an accreditation process. Amending a law which regulates the use of medical radiation exposure can be a method of assistance for the implementation of CIG. However, these methods including legal regulation should be applied restrictively as they could hinder a physician in pursuing an active and appropriate clinical strategy when the decision extends even beyond guidelines.

SPECIFIC EFFORTS FOR IMPROVEMENT OF JUSTIFICATIONS IN KOREA

Guidelines for repeated CT examination

In Korea, efforts to reduce unnecessary medical radiation exposure have been continuously made by the Korean Society of Radiology (KSR) and related national organizations (Fig. 2). In 2013, the KSR and Health Insurance Review and Assessment Service (HIRA) carried out a project to develop guidelines on repeated CT examinations for the appropriate management of high-cost radiologic examinations, which could contribute to preventing unnecessary medical exposures through frequent CT examinations and to recommend repeated evaluations based on medical necessity and evidence-based rationale (22). Repeated CT examination is one of the major causes of the increase in radiation exposure particularly in tertiary hospitals.
Fig. 2
Practical efforts for justification of medical radiation exposure in Korea.
jkms-31-S38-g002
Recently, the KSR High-Cost Radiologic Examination Guideline Committee consisting of radiologists from seven different subspecialties explored the causes of repeated CT examinations and referred a report on repeated medical imaging (23), and developed guidelines for CT studies: Chest CT for lung cancer; abdominal CT of routine protocol; abdominopelvic CT for hematuria; cardiac CT; spine and pelvis CT; brain CT of routine protocol; and pediatric CT studies. Repeated CT was defined as repeating CT study of the same anatomic site within 1 month. According to the causes of the repeated examinations, all of the repeated CT examinations can be classified into acceptable or unacceptable examinations (Table 2). Among the acceptable examinations, ‘supplementary’ examinations added to the primary examination, such as dynamic enhancement or 3-dimensional reconstruction, ‘permissible duplicated’ examinations in which the examination was performed due to the low quality of the primary examination, and follow-up examinations are affiliated. To the contrary, unacceptable examinations would include those in which there is a lack of reasonable rationale for the repeated examinations.
Table 2

Classification of repeated CT examinations in Korean guidelines

jkms-31-S38-i002
Classification Definition Examples
Permissible to repeat Supplementary Repeated examination to obtain supplementary information in adjunct to the primary examination Adding dynamic liver CT for the diagnosis of hepatic masses
Permissible duplicated Repeated examination owing to low quality of the primary examination Adding abdominopelvic CT due to thick scan slices (exceeding 8 mm)
Follow-up Repeated examination to check for changes after treatment or operation, including treatment response and complications Adding brain CT to check for changes in intracranial hemorrhage
Impermissible to repeat Unacceptable supplementary Repeated examination to obtain unclear supplementary information or with an unclear reason for follow-up Adding dynamic liver CT following a high-quality liver CT at an outside hospital
Unacceptable duplicated
These guidelines were accredited by each radiologic society of each subspecialty, and applied to several large tertiary hospitals to investigate the actual conditions of the repeated CT examinations including its frequency and the classification of their causes. As a result, it was revealed that the incidence of repeated CT was approximately 15% and that a quarter of the repetition was diminishable (duplications due to sub-optimal quality and unacceptable cause). The KSR has made an effort to implement the guideline widely and have a plan to investigate the change in the conditions following the dissemination of the guideline.

Development of the Korean clinical imaging guideline

The KSR sympathizing with the need for CIGs, constituted a Clinical Practice Guideline Committee in 2012, and has worked together with the National Evidence-based Healthcare Collaborating Agency of Korea (NECA) to develop Korean clinical imaging guidelines (K-CIG) in 2015. These guidelines are expected to help physicians select the most appropriate clinical imaging modality and to improve justification of medical radiation exposure. In addition, these guidelines should reduce inappropriate medical radiation exposure and medical costs. In developing K-CIG, adaptation of previous CIGs including ACR AC was decided as the development strategy. As mentioned above, adaptation of CIGs may be more reasonable than de novo development of CIG as the medical environment in Korea is very different from other countries. As an example, most of the CIGs do not recommend routine chest radiography for a young and healthy patient prior to general anesthesia, but it is not acceptable for Korean physicians because Korea is still an endemic area of pulmonary tuberculosis.
The process of development has been divided into eight stages: development of key questions and key words; searching guidelines; selection seed guidelines; evaluation of seed guidelines; extraction of evidence; drafting recommendations; internal and external validation; and finally, approval of CIG. To search all eligible guidelines, the developing group will search various databases including MEDLINE, EMBASE, Cochrane CENTRAL, KoreaMed, KMbase, Korean studies Information Service System (KISS), National Digital Science Library (NDSL), National Guideline Clearance (NGC), Guidelines International Network (G-I-N), NICE, and SIGN. Moreover, websites of national radiologic societies including the ACR, UK RCR, and Japanese Society of Radiology (JSR) will also be searched to identify and retrieve CIG related guidelines. Thereafter, the developing group will perform an evaluation of the searched guidelines using the Korean Appraisal of Guidelines for Research and Evaluation (K-AGREE) tool, and give recommendations with searched guidelines to the working group. The working groups will then select seed guidelines from the searched guidelines on the basis of the consistency of key questions, acceptance in Korea, and methodological rigor. Then, they will make a draft with evidence tables. A level of evidence criteria will be modified from the methods of the Oxford Centre of Evidence-based Medicine and ACR, and classified into a 5-grade scale from 1 to 5. Working groups will also decide on the grade of the recommendation: A, recommend to do; B, conditionally recommend to do; C, recommend not to do; and I, not recommend. After making drafts, a consensus group will review the drafts and deduce agreements using a modified Delphi method and the working group will refine the drafts according to the agreement of the consensus groups. Lastly, the developing and working groups will decide on the final recommendations and ask external experts, stakeholders and the government to endorse the CIG for external validation.

CONCLUSION

In this review, we have addressed the importance and necessity of CIGs as action plans for justification of medical radiation exposure using examples from other representative countries, and presented the current issues for the development and implementation of CIGs, and its current status in Korea.
Evidence-based CIGs have been developed by several countries and international organizations, but are not yet widely disseminated globally because several difficult issues remain. To overcome them, adaptation of CIGs was introduced as an alternative development method. Most of all, knowledge regarding the appropriate use of radiologic studies necessitating the deployment of CIGs will appear more frequently and concretely in the near future.
In Korea, specific efforts for the improvement of justification have occurred including the development of guidelines for repeated CT examinations and K-CIGs through adaptation of previous CIGs. We expect to advance and maintain these efforts until well-organized, impartial guidelines which more accurately reflect the medical situation in Korea and to which most related physicians and stakeholders can agree are developed.

ACKNOWLEDGMENT

The authors thank Christopher Woo for his advice on English editing.

Notes

Funding This study was co-supported by the National Evidence-based Collaborating Agency (NECA-C-15-003) and the Korean Society of Radiology (NECA-S-15-002).

DISCLOSURE The authors have no potential conflicts of interest or financial ties to disclose.

AUTHOR CONTRIBUTION Design of the article: Jeong WK, Baek JH. Writing first draft: Jeong WK, Baek JH. Revision: Jung SE, Do KH, Yong HS, Kim MJ, Choi M, Lee M, Choi SJ, Jo AJ, Choi JA. Approval of the final manuscript and agreement of submission: all authors.

References

1. Malone J, Guleria R, Craven C, Horton P, Järvinen H, Mayo J, O’Reilly G, Picano E, Remedios D, Le Heron J, et al. Justification of diagnostic medical exposures: some practical issues. Report of an International Atomic Energy Agency Consultation. Br J Radiol. 2012; 85:523–538.
2. International Atomic Energy Agency (IAEA). Report of a consultation on justification of patient exposures in medical imaging. Radiat Prot Dosimetry. 2009; 135:137–144.
3. Malone J, O’Connor U, Faulkner K. SENTINEL Project special initiative: ethical and justification issues in medical radiation protection. Radiat Prot Dosimetry. 2009; 135:69–70.
4. National Research Council (US) Committee on Health Effects of Exposure to Low Levels of Ionizing Radiations. Health effects of exposure to low levels of ionizing radiations: time for reassessment? Washington, D.C.: National Academy Press;1998.
5. International Commission on Radiological Protection. The 2007 Recommendations of the International Commission on Radiological Protection. ICRP publication 103. Ann ICRP. 2007; 37:1–332.
6. Medina LS, Blackmore CC. Evidence-based radiology: review and dissemination. Radiology. 2007; 244:331–336.
7. Sardanelli F, Hunink MG, Gilbert FJ, Di Leo G, Krestin GP. Evidence-based radiology: why and how? Eur Radiol. 2010; 20:1–15.
8. Royal College of Radiologists Working Party. Influence of Royal College of Radiologists’ guidelines on referral from general practice. BMJ. 1993; 306:110–111.
9. Eccles M, Steen N, Grimshaw J, Thomas L, McNamee P, Soutter J, Wilsdon J, Matowe L, Needham G, Gilbert F, et al. Effect of audit and feedback, and reminder messages on primary-care radiology referrals: a randomised trial. Lancet. 2001; 357:1406–1409.
10. Oakeshott P, Kerry SM, Williams JE. Randomized controlled trial of the effect of the Royal College of Radiologists’ guidelines on general practitioners’ referrals for radiographic examination. Br J Gen Pract. 1994; 44:197–200.
11. Hsu J, Brożek JL, Terracciano L, Kreis J, Compalati E, Stein AT, Fiocchi A, Schünemann HJ. Application of GRADE: making evidence-based recommendations about diagnostic tests in clinical practice guidelines. Implement Sci. 2011; 6:62.
12. Gopalakrishna G, Mustafa RA, Davenport C, Scholten RJ, Hyde C, Brozek J, Schünemann HJ, Bossuyt PM, Leeflang MM, Langendam MW. Applying Grading of Recommendations Assessment, Development and Evaluation (GRADE) to diagnostic tests was challenging but doable. J Clin Epidemiol. 2014; 67:760–768.
13. Royal College of Radiologists. iRefer: making the best use of clinical radiology. 2012. accessed on 3 June 2015. Available at https://www.rcr.ac.uk/irefer-making-best-use-clinical-radiology.
14. Oxford Centre for Evidence-based Medicine.Oxford Centre for evidence-based medicine - levels of evidence (March 2009). accessed on 17 June 2015. Available at http://www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/.
15. American College of Radiology. ACR Appropriateness criteria®. accessed on 3 June 2015. Available at http://www.acr.org/Quality-Safety/Appropriateness-Criteria.
16. American College of Radiology. ACR Appropriateness Criteria®: overview. accessed on 8 June 2015. Available at http://www.acr.org/~/media/ACR/Documents/AppCriteria/Overview.pdf.
17. National Decision Support Company. ACR select. Your rules. Your value. accessed on 8 June 2015. Available at http://www.acrselect.org/downloads/productsheet/ACRselect_value-based_imaging.pdf.
18. Moriarity AK, Klochko C, O’Brien M, Halabi S. The effect of clinical decision support for advanced inpatient imaging. J Am Coll Radiol. 2015; 12:358–363.
19. European Society of Radiology. ESR EuroSafe imaging-together for patient safety. accessed on 10 June 2015. Available at http://www.myesr.org/cms/website.php?id=/en/eu_affairs/esr_eurosafe_imaging_campaign.htm.
20. Remedios D, Hierath M, Ashford N, Bezzi M, Cavanagh P, Chateil JF, Grenier P, Simeonov G, Vilgrain V. Imaging referral guidelines in Europe: now and in the future-EC Referral Guidelines Workshop Proceedings. Insights Imaging. 2014; 5:9–13.
21. Malone J, del Rosario-Perez M, Van Bladel L, Jung SE, Holmberg O, Bettmann MA. Clinical imaging guidelines part 2: Risks, benefits, barriers, and solutions. J Am Coll Radiol. 2015; 12:158–165.
22. Health Insurance Review and Assessment Service. The assessment about test application of the guideline for repeated CT and MRI examinations. accessed on 17 June 2015. Available at http://www.hira.or.kr/rc/icenter/study/getReportInfo.do?pgmid=HIRAA030095000000.
23. Kassing P, Duszak R. Repeat medical imaging: a classification system for meaningful policy analysis and research. Reston, VA: Harvey L. Neiman Health Policy Institute;2013. p. 1–7.
TOOLS
ORCID iDs

Woo Kyoung Jeong
https://orcid.org/http://orcid.org/0000-0002-0676-2116

Jung Hwan Baek
https://orcid.org/http://orcid.org/0000-0003-0480-4754

Seung Eun Jung
https://orcid.org/http://orcid.org/0000-0003-0674-5444

Kyung Hyun Do
https://orcid.org/http://orcid.org/0000-0003-1922-4680

Hwan Seok Yong
https://orcid.org/http://orcid.org/0000-0003-0247-8932

Min-Jeong Kim
https://orcid.org/http://orcid.org/0000-0002-2418-8226

Miyoung Choi
https://orcid.org/http://orcid.org/0000-0002-2424-9965

Min Lee
https://orcid.org/http://orcid.org/0000-0002-1077-059X

Sol Ji Choi
https://orcid.org/http://orcid.org/0000-0002-7472-7945

Ae Jeong Jo
https://orcid.org/http://orcid.org/0000-0003-1228-7021

Jin A Choi
https://orcid.org/http://orcid.org/0000-0002-6574-185X

Similar articles