Methodological Quality Appraisal of 27 Korean Guidelines Using a Scoring Guide Based on the AGREE II Instrument and a Web-based Evaluation

Sung-Goo Chang; Dong-Ik Kim; Ein-Soon Shin; Ji-Eun Jang; Ji-Yun Yeon; Yoon-Seong Lee

doi:10.3346/jkms.2016.31.5.682

Abstract

This study evaluated the methodological quality of CPGs using the Korean AGREE II scoring guide and a web-based appraisal system and was conducted by qualified appraisers. A total of 27 Korean CPGs were assessed under 6 domains and 23 items on the AGREE II instrument using the Korean scoring guide. The domain scores of the 27 guidelines were as following: the mean domain score was 82.7% (median 84.7%, ranging from 55.6% to 97.2%) for domain 1 (scope and purpose); 53.4% (median 56.9%, ranging from 11.1% to 95.8%) for domain 2 (stakeholder involvement); 63.0% (median 71.4%, ranging from 13.5% to 90.6%) for domain 3 (rigor of development); 88.9% (median 91.7%, ranging from 58.3% to 100.0%) for domain 4 (clarity of presentation); 30.1% (median 27.1%, ranging from 3.1% to 67.7%) for domain 5 (applicability); and 50.2% (median 58.3%, ranging from 0.0% to 93.8%) for domain 6 (editorial independence). Three domains including scope and purpose, rigor of development, and clarity of presentation were rated at more than 60% of the scaled domain score. Three domains including stakeholder involvement, applicability, and editorial independence were rated at less than 60% of the scaled domain score. Finally, of the 27 guidelines, 18 (66.7%) were rated at more than 60% of the scaled domain score for rigor of development and were categorized as high-quality guidelines.

INTRODUCTION

Many Korean academic societies and clinical research centers have made efforts to develop and disseminate high-quality clinical practice guidelines (CPGs), and guideline developers mainly include an increasing number of interested physicians who participate in workshops to learn guideline development methodologies. However, a study reported that the quality assessment results using an AGREE Instrument (1 2 3 4) for 66 Korean CPGs developed from 2004 to 2009 were not good (5), and an unsatisfactory quality of Korean guidelines has been reported (6).

In this study, 27 Korean CPGs, all of which were developed between 2013 and 2014 with external funding (National Strategic Coordinating Center for Clinical Research, NSCR) and a required evidence-based development methodology, were assessed, and the domain scores were compared to differentiate between high- and poor-quality guidelines. To make a correct judgement and increase the reliability, guideline appraisers learned how to apply a Korean AGREE II scoring guide (7 8) and utilize practical implementation tools, such as a worksheet, to identify explicit elements in detail and to easily find the elements of disagreement among appraisers. A web-based appraisal system was developed in 2012 and was applied to maximize reliability.

MATERIALS AND METHODS

Guidelines for appraisal

A total of 27 Korean CPGs were assessed under 6 domains and 23 items on the Korean AGREE II instrument using the scoring guide. Of the 27 guidelines, 20 (74.1%) were developed by 7 clinical research centers for rheumatoid arthritis, stroke, depression, end-stage renal disease, ischemic heart disease, chronic obstructive pulmonary disease and type 2 diabetes over a 2-year period (2013-2014). Of the remaining guidelines, 6 (22.2%) were developed by 4 university hospitals, include Korea, Yonsei, Cha, and Chonnam National University Hwasun. One guideline was developed by a Hospital Nurses Association (Appendix 1).

Guideline appraisal process

There were two steps to the appraisal process. First, each guideline was assessed by 4 appraisers who were qualified guideline evaluation members of the Executive Committee for CPGs, Korean Academy of Medical Sciences (KAMS), to increase the reliability of the assessment. Second, 1 senior appraiser made summary statements on 23 items after adjusting for disagreement (Fig. 1). Disagreement was defined as more than 4 judgement score differences among appraisers on the same item using a 7-point rating scale. A score of 1 (strongly disagree) should be given when there is no information or if the concept is very poorly reported, and a score of 7 (strongly agree) should be given when the full criteria for ‘how to rate’ and/or ‘further considerations’ articulated in the user’s manual are met. For the level of judgement between 2 and 6, a score is assigned depending on the completeness and quality of the reporting. Scores increase as more criteria are met and considerations are addressed.

Fig. 1

A framework for the systematic web-based quality appraisal of CPGs in Korea.

A total of 37 qualified appraisers participated in the first-step evaluation of 27 CPGs over 3 months (September to November 2014) and an average of 2.9 (range 1 to 5) guidelines were assessed by each appraiser. During the second-step evaluation, 11 senior appraisers assessed an average of 2.5 (range 1 to 4) guidelines each over another 2-month period (December 2014 to January 2015).

AGREE II scoring guide

To reduce differences among evaluators on the assessment of the quality of the CPGs, a scoring guide, consisting of ninety-two criteria for anchor points 1, 3, 5, and 7 relating to 23 items on the AGREE II instrument, was developed by the Executive Committee for CPGs, Korean Academy of Medical Sciences, Korea. It was based on the ‘user’s manual’ and ‘how to rate’ description in the AGREE II instrument and was rated through a Delphi consensus process.

Providing an education program for appraisers

An education program was provided to 58 physician participants to assess the quality of CPGs developed over a 2-year period (2013 to 2014). A series of courses included how to rate and apply a developed AGREE II scoring guide in Korea.

Web-based appraisal system

A web-based appraisal system from the Korean Medical Guideline Information Center (http://www.guideline.or.kr) was applied. To develop the web-based appraisal system, 3 CPGs were evaluated using the system. After the pre-test was conducted, revisions were made based on the comments from the internal review panels.

Analysis

Domain scores were calculated by summing all of the scores of individual items in a domain and by scaling the total as a percentage of the maximum possible score for that domain. Guidelines that were rated as at least 60% of the scaled domain score for rigor of development were categorized as high-quality guidelines.

RESULTS

The domain scores of the 27 guidelines were as follows (Table 1): the mean domain score was 82.7% (median 84.7%, ranging from 55.6% to 97.2%) for domain 1 (scope and purpose); 53.4% (median 56.9%, ranging from 11.1% to 95.8%) for domain 2 (stakeholder involvement); 63.0% (median 71.4%, ranging from 13.5% to 90.6%) for domain 3 (rigor of development); 88.9% (median 91.7%, ranging from 58.3% to 100.0%) for domain 4 (clarity of presentation); 30.1% (median 27.1%, ranging from 3.1% to 67.7%) for domain 5 (applicability); and 50.2% (median 58.3%, ranging from 0.0% to 93.8%) for domain 6 (editorial independence).

Table 1

Scaled domain percentages of 27 Korean CPGs

CPG No.	Mean scaled domain percentages of 4 appraisers (%)
CPG No.	Scope and purpose	Stakeholder involvement	Rigor of development	Clarity of presentation	Applicability	Editorial independence
1	80.6	58.3	37.0	72.2	44.8	70.8
2	88.9	59.7	76.0	91.7	27.1	58.3
3	80.6	62.5	77.1	84.7	36.5	58.3
4	93.1	48.6	72.4	91.7	21.9	62.5
5	93.1	58.3	90.6	93.1	47.9	93.8
6	83.3	76.4	76.0	98.6	24.0	66.7
7	81.9	55.6	71.4	88.9	26.0	79.2
8	69.4	66.7	76.0	91.7	12.5	64.6
9	63.9	44.4	38.0	88.9	20.8	47.9
10	86.1	62.5	45.3	80.6	22.9	68.8
11	69.4	55.6	71.4	84.7	14.6	77.1
12	94.4	56.9	69.8	75.0	39.6	58.3
13	95.8	70.8	67.2	98.6	27.1	22.9
14	91.7	51.4	72.9	90.3	22.9	66.7
15	97.2	51.4	82.3	97.2	28.1	93.8
16	84.7	58.3	62.5	86.1	18.8	35.4
17	69.4	15.3	15.6	83.3	3.1	0.0
18	61.1	11.1	13.5	58.3	4.2	0.0
19	84.7	52.8	41.1	90.3	55.2	20.8
20	91.7	56.9	49.5	91.7	54.2	18.8
21	97.2	65.3	79.7	95.8	33.3	33.3
22	94.4	63.9	90.1	98.6	40.6	50.0
23	66.7	48.6	61.5	94.4	47.9	20.8
24	97.2	55.6	90.6	100.0	67.7	83.3
25	94.4	95.8	90.6	95.8	27.1	83.3
26	55.6	16.7	29.7	91.7	16.7	6.3
27	66.7	22.2	54.2	86.1	26.0	12.5

CPGs, clinical practice guidelines.

Three domains including scope and purpose, rigor of development, and clarity of presentation were rated at more than 60% of the scaled domain score. And three domains including stakeholder involvement, applicability, and editorial independence were rated at less than 60% of the scaled domain score. Finally, of the 27 guidelines, 18 (66.7%) were rated at more than 60% of the scaled domain score for rigor of development and were categorized as high-quality guidelines (Table 2).

Table 2

Quality appraisal of 27 Korean CPGs using AGREE II

AGREE II domain	60% ≥		60% <		Total
AGREE II domain	No.	%	No.	%	No.	%
Scope and purpose	26	96.3	1	3.7	27	100
Stakeholder involvement	8	29.6	19	70.4	27	100
Rigor of development	18	66.7	9	33.3	27	100
Clarity of presentation	26	96.3	1	3.7	27	100
Applicability	1	3.7	26	96.3	27	100
Editorial independence	12	44.4	15	55.6	27	100

CPGs, clinical practice guidelines; AGREE II, Appraisal of Guidelines for Research & Evaluation II.

DISCUSSION

An appraisal of 27 Korean CPGs was performed using the AGREE II. Overall, the guidelines scored highest on clarity of presentation, with a mean score was of 88.9% (median 91.7%) and lowest on applicability, with a mean score was of 30.1% (median 27.1%). The mean scores of CPGs developed by WHO were higher than those of Korean CPGs for the domains of stakeholder involvement, rigor of development, applicability, and editorial independence (Table 3) (9).

Table 3

Comparison of the mean scaled domain percentages between WHO and Korean CPGs

AGREE II domain	Mean scores of WHO guidelines^* (%)		27 Korean CPGs (%)
AGREE II domain	Pre GRC^† (n = 10)	Post GRC^† (n = 10)	Mean ± SD (Median)	Range
Scope and purpose	62.2	80.4	82.7 ± 12.79 (84.7)	55.6 to 97.2
Stakeholder involvement	49.8	61.2	53.4 ± 18.68 (56.9)	11.1 to 95.8
Rigor of development	30.7	68.3	63.0 ± 22.30 (71.4)	13.5 to 90.6
Clarity of presentation	60.9	78.2	88.9 ± 9.24 (91.7)	58.3 to 100.0
Applicability	49.1	61.6	30.1 ± 15.62 (27.1)	3.1 to 67.7
editorial independence	20.9	73.6	50.2 ± 28.95 (58.3)	0.0 to 93.8

CPGs, clinical practice guidelines; AGREE II, Appraisal of Guidelines for Research & Evaluation II.

*Source: Sinclair D et al. (9), ^†GRC, guideline review committee.

Three domains including stakeholder involvement, applicability, and editorial independence showed less than 60% of the scaled domain score. The mean score of stakeholder involvement in our study was 50.2% (median 58.3%). Reasons for this mean score of less than 60% include no qualified data regarding the views and preferences of patients and/or the public in Korea; no guidelines that consider the patient's values and preferences in Korea; appraisers who did not accurately understand the definition of the views and preferences of the target population (patients, public, etc.); and appraisers who did not identify the proper elements. Tudor et al. (10) reported the lowest score in stakeholder involvement (19%).

Our finding showed that the mean score on the applicability domain was low. Korea has a relatively short history in the development and use of CPGs (7). Gagliardi and Brouwers (11) reported that applicability scored lower than all other domains, and the mean and median domain scores for applicability across 137 guidelines published in 2008 or later were 43.6% and 42.0%, respectively. Some reports have given median applicability scores as low as 10.5% (12) and 20.8% (13). Burgers et al. (14) reported that differences in the applicability of the guidelines could not be explained by the variables studied, including care level, scope, type of guideline, year of publication, type of agency, and whether the guideline was produced within a structured and coordinated program.

To increase score for applicability, following criteria should be met and addressed when developing guidelines; describe and consider facilitators and barriers to its application, provide advice and/or tools on how the recommendations can be put into practice, consider the potential resource implications of applying the recommendations, and present monitoring and/or auditing criteria.

The mean score for rigor of development on the 27 Korean CPGs was 63.0% (median 71.4%), and 66.7% of evaluated guidelines were rated at more than 60% of the scaled domain score for rigor of development and were categorized as high-quality guidelines. Korean guideline developers have learned the elements of high-quality guidelines through workshops. In contrast, Al-Ansary et al. (15) reported low scores for rigor of development (ranging 8.3% to 30% for 9 CPGs).

In general, tailored education programs on the stages of guideline development and practical training on software packages such as Revman5 and GRADEpro will be needed for guideline developers in Korea.

The web-based appraisal system is a very easy and useful e-tool for preparing details about assessment criteria and taking into consideration such tools as the Korean AGREE II scoring guide, and qualified appraisers who have received an appropriate scoring education can accurately assess the good and/or poor quality of CPGs. Furthermore, standardizing and clarifying the evaluation process using e-tool and the Korean scoring guide will be beneficial to guideline appraisers.