PDF
ePub
Citation
Print
INTRODUCTION
The Korean Journal of Urology (KJU), which was created in 1960, became an entirely English-language journal in January 2010 [1]. In addition, after its registration in the Scopus and Excerpta Medica Database (EMBASE), the KJU was indexed in PubMed and PubMed Central (PMC) starting in May 2010, making it an international journal that people worldwide can search and freely download its full text [2]. As the status of the KJU has increased, more prospective clinical studies have been published, and randomized controlled trials (RCTs) have particularly increased in various fields. Because RCTs can avoid bias, they are considered the most reliable method for assessing the effectiveness of therapeutic interventions. RCTs performed incorrectly, however, may produce incorrect results. Therefore, quality assessment of the study should be performed before applying the results of the published study to patients.
The quality assessment of prospective clinical studies includes three methods: individual markers, checklists, and scales [3]. In particular, the assessment standard has a specified form that can allow quantitative assessment and can be easily applied. Thus, it has the advantage of allowing easy comparison of studies. Among the items included in the assessment standard, in particular, three items regarding randomization, double-blinding, and dropout are directly associated with the minimization of bias. The Jadad quality assessment scale (Jadad scale) is a typical quality assessment tool comprising the aforementioned three items [4]. An additional individual index is the assessment of the suitability of allocation concealment. Allocation concealment, which is a separate concept from randomization, which refers to the generation of the order, is described as a method that executes the randomized allocation sequence generated [5]. When patients are assigned for therapeutic intervention, adequate allocation concealment is required to avoid selection bias [6].
In the present study, both quality assessment using the Jadad scale and assessment of the suitability of allocation concealment, which is an individual index, were conducted on RCTs published in the KJU to observe the qualitative growth of design and performance. Quality assessment items with particularly lower scores and factors that may affect the results of the assessment were investigated to provide useful information for further studies in the KJU.
MATERIALS AND METHODS
1. Subjects
A total of 3,516 original articles published in the KJU over the past 20 years from 1991 (volume 32) to 2010 (volume 51) were manually searched.
2. Extraction of randomized controlled trials
Of the prospective studies published in the KJU, those that used terms such as 'randomized', 'randomization', and 'randomly' in the methods session were separately extracted by two researchers. Differences in the results of the extraction were discussed and resolved via third-party arbitration.
3. Quality assessment of the randomized controlled trials extracted
Quality assessment was conducted by using the Jadad scale, and the suitability of allocation concealment was assessed as an individual index. The assessments were conducted independently by two researchers, and the results were compared. Any differences in the results were resolved via third-party arbitration.
1) Jadad scale
The Jadad scale assesses three items regarding the generation of randomization order, double-blinding, and dropout [7]. The range of the total score is 0-5 points. If the total score was 3 points or higher, the study was assessed as being of high quality. If the total score was 2 points or lower, the study was assessed as being of low quality [8]. For studies in which it was impossible to perform double-blinding, the study was assessed as being of high quality if the total score was 2 points or higher [9]. As for the generation of randomization order and double-blinding, if they were only mentioned without a detailed description, one point was assigned to each item. If an appropriate method was described, one more point was added. Meanwhile, if the described method was inappropriate, one point was deducted. Appropriate descriptions of the generation of randomization order included "table of random sampling numbers" and "computer aided generation," whereas inappropriate descriptions included "rotational allocation" and "using medical history number or birthday." As for dropout, if the number and reasons for dropout were described for each subject group, one point was assigned. If there was no dropout, that should also be described in the main section. In that case, there was neither an additional point nor a deduction.
2) Allocation concealment
Descriptions of an adequate concealment method included descriptions such as "non-transparent, sealed bag marked with sequential number," "vessel marked with sequential number," or "randomized allocation controlled by the study center" [10].
4. Presentation of the results of the quality assessment
Annual assessment on the basis of the mean total Jadad score, ratio of high-quality studies, and adequate concealment ratio was conducted by grouping the studies in 5-year periods because of the small number of RCTs extracted. Double-blinding among the items of the Jadad scale analysis was examined except for studies for which double-blinding was impossible. Because a total score of 3 points was given for studies for which double-blinding was impossible, the total score of 3 points was converted into a total score of 5 points for processing.
5. Factors related to the quality of the randomized controlled trials
Three factors associated with the quality of the research papers were also examined: first, intervention with a drug medication; second, support by a research fund or with interventional products from specific institutions; and third, approval of an institutional review board (IRB).
RESULTS
1. Annual quantitative change in published randomized controlled trials
A total of 28 RCTs were published in the KJU. The papers accounted for 0.27% (5 papers) of the journal in the 1990s but increased to 1.34% of the journal (23 papers) in the 2000s (Table 1).
2. Annual qualitative change in published randomized controlled trials
1) Jadad quality assessment scale
The mean Jadad scale was 1.67 points during 1991 to 1995, 1.50 points during 1996 to 2000, and 2.00 points during 2001 to 2005, but decreased to 1.57 points during 2006 to 2010. The number of the papers assessed as high quality was one paper (20%) in the 1990s and seven papers (30.43%) in the 2000s, which showed a pattern similar to that for the Jadad scale (Table 1).
2) Allocation concealment
Among the RCTs, an appropriate concealment method was described in one paper (3.6%) and an inappropriate concealment method was described in five papers (17.8%). An unclear concealment method was described in 22 papers (78.6%), which accounted for the majority of the articles.
3. Analysis of items on the jadad quality assessment scale
When the three items of the Jadad scale were analyzed in the 28 papers, dropout and randomization were described in the highest percentage of articles (85.7%). In addition, appropriate double-blinding and appropriate reporting of withdrawals were described in 11 (61.1%) and 10 (35.7%) articles, respectively. The low total score for the quality assessment was mainly attributable to the small number of appropriate descriptions of dropout and the lack of double-blinding. Among such cases, a total of 10 papers (35.7%) were shown to not be eligible for double-blinding. Thus, the analysis of the double-blinding item was conducted excluding those 10 articles (Table 2).
4. Analysis of factors related to the quality of the articles
During the period of 1980-2010, 21 articles were shown to be associated with a drug intervention, and 7 articles were associated with nondrug interventions such as procedures and surgery. The mean Jadad scale was higher in the papers associated with a drug intervention than in the papers associated with nondrug interventions (1.86 vs 1.43). An intervention with a drug was described in nine papers with high quality (42.9%), which was also higher than for the nondrug interventions. Eleven papers had received external funding, and their mean Jadad scale was 2.09, which was higher than the mean value of 1.53 in the unsupported papers. High quality (54.5%) was shown in the papers with support, and this percentage was higher than in the papers without support. IRB approval was described in six papers. Their mean Jadad scale was 2.33 (range, 1 to 3), which was higher than the mean value for the total RCTs. High quality was shown in five papers (83.3%) (Table 3).
DISCUSSION
There are several quality assessment tools for RCTs, including the Campell, Moher, Chalmers, and Jadad methods, of which the Chalmers and Jadad methods have been the most widely used. In the quality assessment of RCTs, the suitability of concealment for random allocation, double-blinding, and the follow-up rate of subjects are the three most important factors [11]. The Jadad scale, which includes the aforementioned three factors, is a simple and objective tool that is also used for the quality assessment of individual clinical studies for meta-analysis in the Cochrane Database of Systematic Review. Accordingly, the Jadad method was applied for the quality assessment of RCTs in this study. This is the first report of the quality assessment of original articles published in the KJU.
A quality assessment of RCTs published in other journals in Korea was previously conducted in only two studies. Kim et al analyzed 12,760 original articles in five journals published in Korea (The Korean Journal of Internal Medicine, Journal of the Korean Surgical Society, Korean Journal of Obstetrics and Gynecology, Korean Journal of Pediatrics, Korean Journal of Family Medicine) [12]. In that study, no articles with a total Jadad scale of 4 points or higher were found during the period of 1980 to 2000. The number of papers with a score of 2 points increased in the 1990s compared with the 1980s. In addition, the number of papers with a score of 3 points increased from two to seven. In another study, Chung et al conducted a quality assessment of RCTs from 1,290 original articles published in the Korean Journal of Family Medicine from 1980 to 2005 [13]. They reported that RCTs quantitatively increased from 1.09% of the original articles in the 1980s to 2.63% in the 2000s and that the Jadad scale increased from one point in the 1980s to 2.17 points in the 2000s. As for quality assessment in urology, Autorino et al extracted RCTs from abstracts presented at the World Congress of Endourology (WCE) annual meeting during 2004-2006 and assessed them according to the Consolidated Standards for the Reporting of Trials (CONSORT) statement [14]. They reported that RCT was described in the title of the abstracts in most cases, that RCTs accounted for a small portion of the total abstracts, and that only a small number of abstracts satisfied the CONSORT criteria.
In the present study, prospective RCTs gradually increased quantitatively and qualitatively from KJU volume 32 in 1991 to volume 51 in 2010. The number of randomized clinical studies was 4-fold higher in the 2000s than in the 1990s, and the papers with high quality also were shown to substantially increase. An appropriate concealment method was observed in only one paper since 2000. In addition, no description of dropout and a lack of double-blinding were the most common reasons for a low total score. Intervention with a drug, support from specific institutions, and the approval of an IRB were shown to be associated with the quality of the RCTs. In the case of intervention with a drug, easier conduct of double-blinding by use of a placebo could have been possible, and the rate of appropriate double-blinding may have been higher because of the increased number of studies that could perform double-blinding. However, the description of appropriate randomization and dropout was not high in RCTs with a drug intervention, and the reasons for this are unclear. In the case of support from specific institutions, a qualitative increase was shown to be associated with both a high rate of appropriate double-blinding and randomization. This result is likely attributable to the fact that it was possible to design a large-scale study because of the financial support. In addition, two papers approved by an IRB were shown to have high quality in most items. Papers approved by an IRB are meaningful in that the design and conduct of the study are validated during the study planning stage. In particular, in the case of RCTs, IRB review is considered an international standard. However, such ethical regulations were poorly recognized in Korea in the past. At present, because most domestic institutions have an operating IRB or have plans to establish an IRB, IRB review is expected to be helpful for the qualitative improvement of domestic studies.
In foreign studies, when an analysis was conducted on articles published in four major journals (British Medical Journal, The Journal of the American Medical Association, Lancet, New England Journal of Medicine) from January 2000 to December 2000, 46% were shown to have an inappropriate or unclear concealment method [15], but no analysis of the KJU has been reported to date. The KJU, which has been indexed in PubMed and PMC, has published papers with high quality and is making efforts to be registered to SCI(E) [16]. The results of this study can be expected to help researchers of the Korean Urological Association understand and appropriately conduct double-blinding and allocation concealment, which will lead to the publication of research papers with improved quality in the KJU.
CONCLUSIONS
RCTs published in the KJU have shown both gradual quantitative and qualitative growth from 1990 to 2010. However, further efforts are required, particularly for adequate double-blinding and allocation concealment methods, because the growth is insufficient. In addition, more attention should be paid to the establishment of more supportive environments and the acquisition of IRB approval to improve study quality.
XML Download