Meta-analysis is defined as ‘an objective statistical analysis which integrates the diverse results of several independent clinical trials with similar concerns’ [1]. Meta-analysis proceeds through searching and selecting studies, retrieving and coding data, accumulating comparable findings, analyzing the result distribution, and reporting the result [2]. Meta-analysis can estimate not only the direction but also the magnitude of effect size of the result. With meta-analysis, clinicians can obtain the conclusions of evidence-based medicine through the examined study according to various methods [3]. Borenstein et al. [3] reported that meta-analysis, with the property of post-hoc analysis, can reveal the trend of the present study and suggest future research directions. Numerous studies have been performed to evaluate the quality of meta-analyses and systematic reviews following the release of Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist reporting guidelines [4567]. Many researchers have studied and reported on the quality of meta-analyses in the fields of pharmacology and medicine. Mulrow [4] reported that there were no studies that completed the checklists among the 50 meta-analyses published in the medical journals between 1986 and 1987. Sacks et al. [5] examined the validity of 83 meta-analyses and reported that the overall quality of the studies was low. A follow-up study in 1996 revealed no improvement in the quality of meta-analyses [8]. However, there have been few studies reporting on the methodological quality of meta-analyses and systematic reviews despite the enormous number of studies using meta-analytic techniques in the field of anesthesia. There are limitations in assessing the quality of meta-analyses and systematic reviews, and it is difficult to distinguish good-from poor-quality reviews because of the wide range of quality. There has been a continued proliferation of scales and checklists for assessing the quality of meta-analyses and systematic reviews [91011]. Well-designed guidelines will assist many authors in assessing the methodological quality and integrating the results of their meta-analyses and systematic reviews. The practice of evidence-based medicine will be further established by high-quality evidence from meta-analyses and systematic reviews. With evidence-based medicine, clinician can make the best decisions. These trials would allow researchers to improve the level of quality.

The purpose of this study was to evaluate the reporting and methodologic quality of the meta-analyses and systematic reviews according to the AMSTAR and PRISMA guidelines in the anesthesia literature, as well as the participation of statisticians as authors. We assessed the rate of a ‘Yes’ reply to the various items in the title, abstract, introduction, materials and methods, results, and discussions of the meta-analyses published. We would like to suggest basic data to improve the quality of studies.

A search was conducted to identify all meta-analyses and systematic reviews published in the British Journal of Anaesthesia (BJA), Anaesthesia, and Korean Journal of Anesthesiology (KJA) between Jan. 01, 2004 and Nov. 31, 2016. The words ‘systematic review,’ ‘meta-analysis,’ ‘meta analysis,’ and ‘meta analyses’ were searched either in the title, abstract, or key words. The search was conducted on MEDLINE, Scopus, and KoreaMed, and the results were cross-referenced with searches performed on the web sites of the respective journals.

We attempted to apply the AMSTAR and PRISMA checklists to all meta-analyses selected according to our criteria, and a score was assigned to each paper. AMSTAR is an eleven-item checklist used to assess the quality of methodology for meta-analyses and systematic reviews (Appendix 1). The creators of AMSTAR proposed three answer options for each item: ‘yes,’ ‘no,’ and ‘not applicable.’ To obtain a ‘yes’ for an item, the meta-analysis must contain all the major components within that item. AMSTAR can be scored both on the basis of individual components and as a checklist by summing the scores (overall score). PRISMA is a checklist for reporting that has 27 entries divided into 7 sections (Appendix 2). We recorded one of three answers for each item: ‘yes,’ ‘no,’ or ‘not applicable.’ Some of the items in the checklist contain multiple components, so if at least half of them were met, then that item received a ‘yes.’ If less than half of the components were met or a key component was lacking, the item was given a ‘no.’ AMSTAR and PRISMA checklists were made by checking the descriptions of items and sub-items of the title, abstract, methods, results, and discussion. If descriptions of items and sub-items were incomplete, we coded partially described items as ‘1’ or ‘0’ depending on whether the meaning of items (sub-items) was described correctly or not. We shared the concept of the items (sub-items) before analyzing the entire study. We obtained inter-rater reliability after separate analysis of 10% of a study by an independent researcher. We repeated this procedure until we obtained a reliability of 85% for each item (sub-items). We investigated the influence of participation of statistician on the quality of meta-analyses. The participation of statisticians was confirmed by checking authors' affiliations.

The statistical process was carried out using Jandel Sigma-Stat® (version 4.0, Systat Software Inc., San Jose, CA, USA) and SPSS version 24.0 (IBM Corp., Armonk, NY, USA) for Windows. The normality of the categorical data was tested, and then the data were analyzed using the Mann-Whitney rank sum method and the Wilcoxon signed rank test. The significance level was P < 0.05 in all cases.

We identified total 121 meta-analyses published in the BJA, Anaesthesia, and KJA from January 2004 through the end of November 2016 using our inclusion criteria. BJA published 75 meta-analyses (Table 1). Anaesthesia and KJA published 43 and 3 meta-analyses, respectively (Table 1). The most common article topics in all journals were drugs. More than half of the meta-analyses were generated from data retrieved from randomized controlled trials (RCTs).

AMSTAR and PRISMA checklist items and sub-items are described in the Appendix (Appendixes 1 and 2). The results for AMSTAR and PRISMA checklists for articles published in the BJA, Anaesthesia, and KJA from January 2004 through the end of November 2016 are shown in Figs. 1 and 2. The percentage of ‘Yes’ response to the items in the AMSTAR and PRISMA checklists is shown in Tables 1 and 2, respectively. The percentage of ‘Yes’ replies to 27 items (sub-items) of the PRISMA guidelines checked in meta-analyses published after the year 2010 was significantly increased compared to that of studies published before the year 2009 (P = 0.014 for Anaesthesia, Table 2). For the AMSTAR checklist, over 50% of the papers received a positive response for items (Q2, Q3, Q5–Q11 in the BJA; Q1–Q3, Q5–Q11 in the Anaesthesia; Q1, Q3, Q5–Q10 in the KJA) (Fig. 1, Table 3). Most papers clearly stated their research question and search criteria of the AMSTAR (Q3, Q5–Q10) (Table 3). However, there was a very low response regarding the status of publications used as an inclusion criterion (Q4) of the articles published in the BJA, Anaesthesia, and KJA and conflict of interest (Q11) in the KJA (Table 3). Publication bias was observed in 62.4% of the papers in the BJA, 62.8% in the Anaesthesia, and 100% in the KJA (Table 3). Conflict of interest is described in 54.5% of the papers in the BJA, 53.5% in the Anaesthesia (Table 3). Overall fulfillment rates of the PRISMA checklist in the BJA, Anaesthesia, and KJA were 85.1, 81.3, and 87.7%, respectively (Fig. 2, Table 4). For the PRISMA checklist, over 50% of the papers received a positive response for items (Q1–Q11, Q13, Q14, Q16–Q18, Q20, Q21, Q23–Q27 in the BJA; Q1–Q11, Q13, Q14, Q16–Q18, Q20, Q21, Q24–Q26 in the Anaesthesia; Q1–Q11, Q13–Q18, Q20–Q26 in the KJA) (Fig. 2, Table 4). Most papers clearly stated their research question and inclusion criteria of the PRISMA (Q1–Q4, Q6–Q11, Q13, Q14, Q16–Q18, Q20, Q21, Q24–Q26) (Table 4). However, there was a very low response for risk of bias in individual studies (Q12) of the articles published in the BJA (44.0%), Anaesthesia (39.5%), and KJA (33.3%) (Table 4). Complete responses were recorded for the item of title in PRISMA that required using ‘meta-analysis’ in the Anaesthesia and KJA. Analysis of the title, abstract, and introduction of the study in almost all papers (over 90%) identified the report as a systematic review, meta-analysis, or both and demonstrated that a structured summary was provided in the abstract. Numerous papers (66.7%, 51.2%, and 66.7% of those in the BJA, Anaesthesia, and KJA, respectively) indicated that a protocol was registered and accessible (Table 4). All of the trials included in this study provided an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). The methods of additional analysis (Q16) such as subgroup analysis, sensitivity analysis, and meta-regression were described in 76.0%, 58.1%, and 66.7% of the papers in the BJA, Anaesthesia, and KJA, respectively (Table 4). The results of additional analysis (Q23), were described in 77.3%, 48.8%, and 100% of the papers in the BJA, Anaesthesia, and KJA, respectively (Table 4). Regarding the discussions, almost all papers provided good descriptions of the summary of evidence (Q24), limitations (Q25), and conclusions of the study (Q26) (Table 4). Meta-analyses of the Anaesthesia did not describe their funding sources (Q27) well, and these were not described in the KJA at all (Table 4).

κ-coefficient analysis was utilized to evaluate interreviewer agreement. The κ value for inter-rater agreement was 0.86 (95% CI, 0.78 to 0.93), demonstrating a high reliability between reviewers. There were no inter-rater disagreements during the review process. The influence of participation of statisticians significantly contributed to improved reporting quality in all the meta-analyses in the included anesthesia literature (P = 0.004, Table 5). In the BJA, participation of statistician as authors statistically improved average score of the PRISMA items (subitems) assessed in anesthesia meta-analysis literature published from Jan. 2004 to Nov. 2016 (P = 0.003, Table 5).

Meta-analysis is an effective statistical method to obtain a major decision regarding trials with similar concerns. It is also effective in cases in which there is no time to collect raw data, as well as in saving costs and labor. Meta-analysis has become popular in the medical literature. Remarkable advances in evidence-based medicine have been responsible for the development of checklists for reporting and methodology of meta-analyses and systematic reviews. With the development of methodology by different research groups, the need for modern statistical analysis for increasing academic achievements has contributed to successful agreement regarding meta-analysis.

We have investigated all the meta-analyses published in the anesthesia literature (BJA, Anaesthesia, and KJA) from Jan. 2004 to Nov. 2016. Initially, we attempted to evaluate and compare the reporting and methodologic quality of the meta-analyses and systematic reviews published in the KJA and major anesthesia literature. However, the original plan was modified to compare the quality of the meta-analyses and systematic reviews published in the BJA, Anaesthesia, and KJA as a preliminary study because of an enormous amount of information in the major anesthetic journals compared to that of the KJA. Regarding the title and abstract, almost all of the studies clearly stated meta-analysis and systematic review, with the exception of 6 papers (8.0%) in the BJA. The abstract was well organized, but the type of analytic model was not stated clearly in the study. The author's selection of type of analysis model depends on the nature of the data. Recently, the advantage of the random-effect model was described and recommended by several investigators, but this model was not adopted widely because of the complexity of its calculation and underestimated recognition [12]. Although the studies were selected for this review with our criteria, meta-analysis and systematic review was not stated clearly in the title in several studies. There was one meta-regression [13] and study of review articles that was not accounted for statistically because of poor standardization and heterogeneity [1415]. Systematic review is a review of a formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and collect and analyze data from the study. However, meta-analysis is a statistical method that may or may not be used to analyze and summarize the results of included studies. Meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies [16]. Precise estimation of the direction and amount of effect size of the intervention is the most characteristic advantage of meta-analysis. Therefore, it is advisable to consider another terminology, such as ‘qualitative meta-analysis,’ instead of meta-analysis or systematic review, in cases of studies in which the study results were not analyzed using statistical techniques. The reader can obtain critical and updated information to understand the magnitude of the study process and results of meta-analysis from the abstract. Therefore, structured information regarding the type of model used in the analysis, the origin of data, and quality of the study (Q15, Q19, and Q22 of the PRISMA items) should be described in the abstract. Detailed suggestion by structured abstract is contradictory to the limitation of word count in the abstract.

The main topic suggested in the introduction may play a critical role in searching and selecting relevant studies. Items of present eligibility criteria (Q6), information sources (Q7), searching strategy (Q8), collecting of grey literature (Q9), and any process for obtaining and confirming data from investigation (Q13, 14) were generally described well in this method. However, items of risk of bias (ROB) (Q12, Q15, Q19) for meta-analysis published in journals were not complied with in this study; that is to say, less than 50% of the requirement for these items was met. The reason for low responses to the item of ROB was due to poor description in the meta-analyses. However, following studies should investigate whether or not the reason for low responses was due to poor examination by the coder.

Methods may affect the study results. Method items in the guideline are important to establish the validity and reliability of the study results. It is crucial to describe whether the author tried to search the present full electronic database. The quality of the meta-analysis is determined by the quality of included studies. In this meta-analysis, certain items (Q5, Q12, Q15, Q19, Q22, and Q27) were not described well in the BJA and Anaesthesia. Items Q12, Q19, and Q27 were poorly described in the KJA. Searching strategy and process for selecting studies were described well in the included studies. The coding sheet of each study is equivalent to one paper of a survey questionnaire. Obtaining data from a questionnaire is a technical process, but the coding process from an individual RCT in meta-analysis is retrieving quantitative results from each study. More than two experts in meta-analysis are necessary for data extraction from reports. Detailed description regarding qualified experts in extracting data from reports and reporting the confidence will increase the validity and reliability of the results of meta-analysis. Description of the third author who coordinates the discrepancy between two coders is necessary to increase the reliability of the process of analysis. Detailed explanation of the management of missing data will increase the objectivity and validity of meta-analysis.

Several studies did neither obtain the mean effect size nor calculation of it. These studies passed over the basic purpose of meta-analysis. The main objective of meta-analysis is to summate the effect size of the result quantitatively, which discriminates meta-analysis from an individual study. To obtain mean effect size, the author should consider sample size. A large sample size can affect determination of effect size and increase the validity of the study. It is advisable to report an effect size with a confidence interval rather than report it as a point estimate. To estimate the mean effect size, it is advisable to evaluate the homogeneity of the individual study. It should be determined whether the result of the study was derived from a homogenous population, and also clarify whether a fixed effect model was used in evaluating homogenous data and a random effect model for heterogenous data [1718]. In general, estimation of effect size will be overestimated when a fixed effect model was used in calculating heterogenous data [1920]. Almost all of the studies provided good information regarding selection of the model type, description of the analytic mode, and evaluation of the homogeneity. To explain the process of meta-analysis objectively, the investigator should clearly report the exclusion and inclusion criteria, and this information will contribute to the validity of the study. It will increase the reliability of the study when a follow-up researcher repeats the same meta-analysis and obtains a similar result. If an author reports a table that shows an effect size and sample size of the included study together, the whole process of estimating the effect size is clearly revealed. Therefore, these processes will increase the validity and reliability of the results of meta-analysis. With the analysis of our results, description of ROB within studies [Q19; BJA (38.7%), Anaesthesia (41.9%), KJA (33.3%)], ROB across studies [Q22; BJA (41.3%), Anaesthesia (46.5%)], and additional analysis [Q23; BJA (77.3%), Anaesthesia (48.8%)] regarding whether the author performed a sensitivity analysis, subgroup analysis, and meta-regression to evaluate the heterogeneity of the included studies according to the guidelines of AMSTAR and PRISMA was poor. However, description of the items 22 and 23 in the KJA was perfect. Items of objectives (Q4), information sources (Q7), effect size of the individual study, sample size, table of the characteristic of the study (Q18), and summary of evidence (Q24) were perfectly compliant in the BJA, Anaesthesia, and KJA.

In the discussions, the explanation of study characteristics describing whether the meta-analysis is the first trial or not should be described. The limitations of previous studies and the logic of the present study also should be described. All of the included articles in our study described summary of the evidence (Q24), limitation (Q25), and conclusion of the study (Q26) well.

We investigated the influence of participation of statisticians on the reporting and methodologic quality of meta-analysis. It has been argued that the creation of a meta-analysis requires a team, which should include both statisticians and physicians. Meta-analyses are also of high quality when the statistician focused on designing and conducting the study [21]. In our study, average scores of AMSTAR items of a meta-analysis in which a statistician was not part of the team did not significantly differ from those of teams of physicians alone (Table 6). However, average scores of PRISMA items of a meta-analysis in the anesthesia literature as a whole and in the BJA in which a statistician participated as part of a team were significantly higher than those of teams of physicians alone (P = 0.004, P = 0.003 respectively, Table 5). The relative discrepancy between the scores in this study comes from the different aims of the checklists (Appendixes 1 and 2).

AMSTAR was designed to reveal methodological quality, and the PRISMA statement is related to reporting quality. AMSTAR guidelines were developed in 2007 by combining elements of the Overview of Quality Assessment Questionnaire (OQAQ) and the Sacks instrument as well as other items based on methodological advances that had been made since the OQAQ and Sacks had been introduced [5]. The first checklist specific to meta-analyses was the Quality of Reporting of Meta-analyses (QUOROM), which was published in 1999 and designed to address the suboptimal reporting of meta-analyses [22]. QUOROM was revised to the PRISMA in 2009 to encompass both meta-analyses and systematic reviews [16]. PRISMA is a checklist for reporting that has 27 items divided into 7 sections: title, abstract, introduction, methods, results, discussion, and funding. Since their development, AMSTAR and PRISMA have become widely accepted as tools to ensure adequate reporting and methodology of meta-analyses and systematic reviews. Although scientific proof supports only a subset of the items, we suggest that these guidelines should be used as a standard in the preparation, reporting, and appraisal of meta-analyses of RCTs. As of this date, AMSTAR is widely used by investigators in the medical fields, as well as policy-making associations. AMSTAR has gained a wide acceptance in reliability, respectability, and reproducibility. The purpose of the PRISMA statement is to help authors improve the reporting of meta-analyses and systematic reviews. It may be used as a basis for evaluations of interventions. PRISMA may also be useful for critical appraisal of published systematic reviews. However, the PRISMA checklist is not a quality assessment tool to estimate the quality of a systematic review.

Limitations of our study are that this study was limited to journals related to anesthesia. Meta-analysis of anesthesia journals has been considered to be of higher quality than studies published in other specialty journals [2324]. Secondly, we evaluated whether participation of statisticians contributed to the reporting and methodology quality. However, we did not investigate the previous experience with meta-analysis of the corresponding author at all. It should be investigated whether participation of an expert in meta-analysis will increase the quality of the study or not.

In conclusion, even though there is little variability in the methodology of meta-analysis in the anesthesia literature, we have proved that the quality of meta-analysis in the anesthesia literature seems to be improving in reporting with time after the release of the AMSTAR and PRISMA guidelines and the participation of statisticians. We suggest that reporting and methodology items should be used for every submitted meta-analysis to increase the quality of the process of evidence-based documents. Applying AMSTAR and PRISMA to future meta-analyses may result in higher quality in both reporting and methodology. We strongly recommend the use of clear guidelines of both AMSTAR and PRISMA for the authors, and the reviewers should be familiar with the checklists accordingly.