Journal List > Ann Rehabil Med > v.44(2) > 1150850

Han, Kang, Lee, Chung, and Lee: A Bibliometric Analysis Using Alternative Metrics for Articles in the Annals of Rehabilitation Medicine

Abstract

Objective

To investigate the articles in the Annals of Rehabilitation Medicine (ARM) using a bibliometric analysis to verify whether there is a correlation between the topics of interest for expert groups and the public media.

Methods

A total of 1,088 ARM articles from the third issue of 2011 to the third issue of 2019 were analyzed. We conducted a bibliometric analysis of the articles using conventional metrics (CM) and alternative metrics (AM). The CM was investigated by collating the type of publication, number of citations, and the specific field of rehabilitation medicine for each article. The AM was analyzed using the Altmetric Attention Score (AAS) provided by Altmetric, the leading AM company. The correlation between the number of citations and the AAS was tested using the Spearman rank correlation coefficient.

Results

The combined ratio of original articles and case reports was over 90% in this study; however, the total distribution was significantly different compared to previous bibliometric studies (p<0.05). There were 233 articles that satisfied both conditions of at least one citation and at least one AAS point. The number of citations and the AAS were found to have a statistically significant positive linear correlation on a scatter plot (r=0.216, p=0.001).

Conclusion

There is a significant correlation between AM and CM, which means itis important to increase the dissemination of academic knowledge through the public media and increase the status of the journal by increasing the citation-related index.

INTRODUCTION

Bibliometrics is a form of research that combines the traditional bibliographic research of analyzing the information obtained from published materials with a technical method to quantitatively calculate the impact of the publications [1]. Recently, the number of global reports created by individuals in the medical field utilizing bibliometrics has increased [2].
Since the concept of the Impact Factor (IF), which quantifies the degree of citations within an expert group was first introduced in 1975, the indices for journals listed in the Journal Citation Reports (JCR) have been calculated annually. The IF is now the most objective indicator when referring to the conventional metrics (CM) of bibliometrics [3]. However, with the development of innovative mobile devices and platforms, there is a rapidly growing demand for real-time quantitative measurements for the public dissemination of scientific knowledge through various media. In response to this real-time public demand, using alternative metrics (AM) has been proposed as a new research methodology and as an alternative tool to complement CM [4].
Altmetric LLP, a leading AM company based in London, England [5], has designed specific algorithms to evaluate the public dissemination influence of a scientific article through social media. Using these algorithms, they have developed an Altmetric Attention Score (AAS) for scientific articles through their website (https://www.altmetric.com). The publication of research using the AAS has increased in a variety of medical sciences [6–8].
In particular, as rehabilitation medicine manages both acute and chronic therapeutic care, a multidisciplinary treatment approach that encourages patients and their families to participate in the long-term recovery phase is essential. It requires the rehabilitation medicine necessary for synchronizing insights between medical professional groups and the public. Nevertheless, there have only been a few studies that have investigated international rehabilitation-related journals using alternative bibliometrics [9–11].
The Korean Academy of Rehabilitation Medicine, the representative academic society for rehabilitation medicine in the Republic of Korea, began publishing an official journal in 1977 and has regularly published it six times a year based on an English-only policy since 2011 [12]. The Annals of Rehabilitation Medicine (ARM) is the name of this journal and several bibliometric studies have been published in it with respect to overall compositions [13], statistical methods [14], research design [15], publication ratio of abstracts presented at the annual meeting of the society [16], and the status of ARM at the international level [12]. However, there have been no studies using AM methodologies in ARM.
This study aims to investigate ARM articles using a bibliometric analysis that includes conventional and AM and to verify whether there is a correlation between the topics of interest between expert groups and the public media.

MATERIALS AND METHODS

The ARM website (https://www.e-arm.org) lists the articles published in each issue and offers the ability to search these articles through open-access. To analyze the bibliometrics that correlate with international relevance, we included only ARM articles published since the third issue of 2011, when the society introduced an English-only policy.
This study simultaneously adopted both CM and AM methodologies for the bibliometric analysis of ARM. The CM was investigated by analyzing the type of publication, number of citations, and the specific field of rehabilitation medicine for each article. The types of publications were divided into four categories: review articles, original articles, case reports, and others. The number of citations per article was verified by using CrossRef’s text services, which are provided in the banner on the ARM website. The specific fields were distinguished as ten subjects, depending on the way in which the academy categorized the affiliated members who were fourth-year resident doctors in the annual examination for the specialized rehabilitation medicine qualifications. The subjects are as follows: “brain and neurorehabilitation”, “neuromuscular rehabilitation and electrodiagnostic medicine”, “pain and musculoskeletal rehabilitation”, “spinal cord injury rehabilitation”, “physical medicine”, “pediatric rehabilitation”, “prosthetics and orthotics”, “geriatrics and cancer rehabilitation”, “cardiopulmonary rehabilitation”, and “sports rehabilitation” [17].
The AM was analyzed using the data provided by Altmetric; their website provides the AAS, which is a weighted score of the total mentions of the article across various online media (Table 1). The score is also provided on the banner of the Altmetric website; however, it is limited to confirming the specific details of the score. Altmetric formally approved this study, and so detailed data could be obtained and analyzed without restriction. The data were collected on a specific day (June 30, 2019) to avoid changes in the online activities of the articles.
Two specialists in rehabilitation medicine, who have both worked at a general hospital for more than 10 years as a trainer for resident doctors independently, conducted a review and analysis for the articles. If these two reviewers disagreed, a consensus was achieved through open discussion. The local ethics committee reviewed and approved an exemption for the study.
All of the statistical analyses were performed using IBM’s SPSS version 19.0 for Windows (IBM, Armonk, NY, USA). The distribution patterns according to the type of ARM publications were compared with previous data and measured at different publication times for the journal using a chi-squared test. The Spearman rank correlation coefficient was used to evaluate the correlation between the number of citations and the AAS; for all the tests, the statistical significance was set at p<0.05.

RESULTS

From the third issue of 2011 to the third issue of 2019, ARM published 1,088 articles, and the distribution pattern for the bibliographic variables of ARM was identified (Table 2). First, with respect to the type of publication, there were 779 original articles, which was the highest percentage (71.6%), and this was followed by the percentage of case reports (24.4%), others (1.0%), and review articles (0.4%) in descending order. From a study of 559 ARM articles from the previous version of the journal that were published from 1977 to 1993, Lee et al. [13] reported that the most common type of publication type was original articles, followed by case reports, review articles, and others in descending order. Comparing the results of that study with this study, there was a significant difference in the distribution pattern of the type of publication (p<0.05) (Fig. 1).
Second, for the specific field of rehabilitation medicine, the number (percentage) of articles regarding brain and neuro-rehabilitation and pain and musculoskeletal rehabilitation was 363 (33.4%) and 263 (24.2%), respectively. The combination of the two fields accounted for more than 50% of all of the articles and they were followed by articles on neuromuscular rehabilitation and electrodiagnostic medicine, pediatric rehabilitation, spinal cord injury rehabilitation, cardiopulmonary rehabilitation, geriatrics and cancer rehabilitation, prosthetics and orthotics, physical medicine, and sports rehabilitation in descending order.
In the bibliometric analysis, the number of articles cited by a journal at least once was 800, and the citations were in the range of 1 to 128 (mean, 5.60±6.22). The number of articles with at least one AAS point was 271, and the AAS was in the range of 1 to 127 (mean, 3.20±8.92) (Table 2). The detailed data that constitutes the AAS were obtained through the Altmetric website, and for all of the 271 articles included there were 1,003 mentions on Twitter (67%), Facebook (14.3%), and Sina Weibo (6.88%), and in patents (6.78%) and the news (2.39%).
There were 233 articles satisfying both conditions of at least one citation and at least one AAS point (Table 2). For those articles, as each citation distribution and AAS deviated to the lower values, the scores were converted to log values for the parametric statistical analysis. The number of citations and the AAS were found to have a statistically significant positive linear correlation on the scatter plot (r=0.216, p=0.001) (Fig. 2).

DISCUSSION

This study was conducted to investigate the ARM articles using a bibliometric analysis involving both conventional and AM. It demonstrated that there was a significant correlation between the citations in academic journals and the mentions on public media.
Since the concept of bibliometrics was introduced into the field of medicine, the IF for CM has been essential for discussions about the journal. Most researchers strive to publish their articles in a journal with a higher IF, and it is common for schools, institutions, and hospitals to use this as an objective criterion to judge performance [3]. However, as the IF is updated only once a year and has a large variation in each academic field, it has limitations. In addition, when the IF is emphasized too much, there could be a distorted phenomenon, and the composition and editing direction of the journal can adjust to it [18]. As a complementary concept, AM has been discussed actively to measure the dissemination of scientific knowledge to a public audience through social media. To evaluate a particular article’s influence, AAS has been created by Altmetric, which is a weighted score of the total mentions of the article across various online media (Table 1). AAS is based on three main domains: authors, sources, and volume. To rate a score, they gather information from diverse online media sources. After putting it all together, the AAS indicates a quantified approximation of all the real-time attention picked up for a research output. The score is useful when looking at several articles together in order to identify the level of online activity related with a particular research output [6–8,11].
Since 1995, when the first bibliometric study was published, various bibliometrics-related research had been published by ARM. In particular, Huh [12] reported that the change of the ARM language policy to English-only was successful in elevating the status of the journal to the international level and subsequently analysis using CM became possible. In line with Huh’s suggestions for meeting international standards, the ARM website currently offers metric data provided by CrossRef. However, there have been no studies that have applied alternate metrics to the bibliometric analysis of ARM articles.
This study investigated the bibliometric analysis using both CM and AM methodologies. First, with respect to the CM, the ratio of original articles was the highest, followed by case reports. The combined ratio of both was over 90%, and this was similar to the results from a previous study [13]. However, when this was examined in detail, original articles were observed to be decreasing, and the ratio of review articles had also reduced remarkably— and this was statistically significant. Considering the findings that more review articles lead to more citations and a higher IF, it is necessary to encourage the submission and publication of review articles [19].
From the CM, for the 10 specific fields of rehabilitation medicine, the ratio of brain and neuro-rehabilitation and pain and musculoskeletal rehabilitation accounted for more than half of the total. Considering that the 10 subjects were selected to assess the minimal qualification by sector of resident doctors completing rehabilitation medicine courses, and the importance of diversity in the field of rehabilitation medicine, there might be a need to review the cause for this biased phenomenon by subject.
For the analysis of the AM of the ARM articles, using the data officially provided by Altmetric, the rate of Twitter mentions was overwhelmingly higher among all of the 14 types of media used in the algorithm for the calculation of AAS. This means that Twitter is the most used form of social media, and from the alternative bibliometric analysis of the journal, its influence in the spread of medical knowledge is increasing. In a study on the AM of the Journal of Physical Medicine and Rehabilitation [10], the authors reported that a social media strategy using Twitter was effective for disseminating their research and creating a larger social discourse. Since 2012, the Korean Academy of Rehabilitation Medicine has had both Twitter (https://twitter.com/karm_1972), and Facebook (https://www.facebook.com/karm.or.kr) accounts. However, they have not been actively operating them to date, and their Twitter account has only two posts and 46 followers (verified on June 30, 2019).
The most important point of this study was the comparison between the AM and the citations of the CM, and there was a significant correlation between the two variables. As in previous studies highlighting the importance of the complementary role of the AM to the CM [7,8,10], our findings imply that the effort to increase the AM could elevate the status of ARM. Therefore, a strategic approach is necessary to carry out alternative bibliometric monitoring continuously on the research achievements of the journal, and to spread the journal’s articles to the public effectively by actively utilizing social media. The view of this study is in contrast to some studies that did not reveal significant correlations between the two, which seems to be caused by a research design or period that does not consider such a comparison [9,11].
The present study was limited by an inherent shortcoming of Altmetric and therefore the validity of the social media mentions was hindered due to the anonymity of online communication. In addition, it was difficult to compare the results of this research with those of other international rehabilitation-related journals that have been granted an IF. In the future, when ARM is also given an IF, it is expected that it would be possible to analyze its bibliometrics and compare these objectively with other journals.
In conclusion, our study demonstrated that for this journal, the AM significantly correlated with the CM, and it is important to actively use social media to increase the AM. It is also important for the academic society to increase the dissemination of academic knowledge through public media, and increase the citation-related index for the status of the journal.

Notes

No potential conflict of interest relevant to this article was reported.

Conceptualization: Han SC, Kang HJ, Lee WJ. Methodology: Han SC, Kang HJ, Lee WJ. Formal analysis: Han SC, Kang HJ, Lee WJ. Project administration: Han SC, Kang HJ, Lee WJ, Chung HS, Lee JH. Visualization: Han SC, Kang HJ, Lee WJ. Writing - original draft: Han SC, Lee WJ. Writing - review and editing: Han SC, Kang HJ, Lee WJ. Approval of final manuscript: all authors.

REFERENCES

1. Bonciu C. Title, abstract and keywords: essential issues in medical bibliographic research. Rev Med Chir Soc Med Nat Iasi. 2005; 109:916–26.
2. Cooper ID. Bibliometrics basics. J Med Libr Assoc. 2015; 103:217–8.
crossref
3. Mathur VP, Sharma A. Impact factor and other standardized measures of journal citation: a perspective. Indian J Dent Res. 2009; 20:81–5.
crossref
4. Baheti AD, Bhargava P. Altmetrics: a measure of social attention toward scientific research. Curr Probl Diagn Radiol. 2017; 46:391–2.
crossref
5. Zahedi Z, Costas R. General discussion of data quality challenges in social media metrics: extensive comparison of four major altmetric data aggregators. PLoS One. 2018; 13:e0197326.
crossref
6. Araujo R, Sorensen AA, Konkiel S, Bloem BR. Top altmetric scores in the Parkinson’s disease literature. J Parkinsons Dis. 2017; 7:81–7.
7. Kim ES, Yoon DY, Kim HJ, Lee K, Kim Y, Bae JS, et al. The most mentioned neuroimaging articles in online media: a bibliometric analysis of the top 100 articles with the highest Altmetric Attention Scores. Acta Radiol. 2019; 60:1680–6.
crossref
8. Kim HJ, Yoon DY, Kim ES, Yun EJ, Jeon HJ, Lee JY, et al. The most mentioned neurointervention articles in online media: a bibliometric analysis of the top 101 articles with the highest altmetric attention scores. J Neurointerv Surg. 2019; 11:528–32.
crossref
9. Marincek C, Franchignoni F. Some thoughts on bibliometrics, usage metrics and altmetrics concerning the International Journal of Rehabilitation Research. Int J Rehabil Res. 2019; 42:193–5.
crossref
10. Niehaus WN, Silver JK, Katz MS. The PM&R Journal implements a social media strategy to disseminate research and track alternative metrics in physical medicine and rehabilitation. PM R. 2018; 10:538–43.
11. Knowlton SE, Paganoni S, Niehaus W, Verduzco-Gutierrez M, Sharma R, Iaccarino MA, et al. Measuring the impact of research using conventional and alternative metrics. Am J Phys Med Rehabil. 2019; 98:331–8.
crossref
12. Huh S. The elevation of Annals of Rehabilitation Medicine to the status of an international journal after adopting an English-only policy. Ann Rehabil Med. 2015; 39:661–6.
13. Lee YH, Jun Sung Park JS, Yi TI, Shin JC. A study of overall compositions and yearly trend of the Journal of Korean Academy of Rehabilitation Medicine for calender years 1977 through 1993. J Korean Acad Rehabil Med. 1995; 19:417–24.
14. Kim JH, Kim JS. Statistical methods used in the Journal of Korean Academy of Rehabilitation Medicine. J Korean Acad Rehabil Med. 1998; 22:46–55.
15. Kim J, Yoon S, Kang JJ, Han K, Kim JM, Kim SK. Research designs and statistical methods trends in the Annals of Rehabilitation Medicine. Ann Rehabil Med. 2017; 41:475–82.
crossref
16. Lim JK, Han JY, Lee HC, Lee J, Chung H, Kim JM, et al. Analysis of publication status of abstracts presented at the annual meeting of the Korean academy of rehabilitation medicine. Ann Rehabil Med. 2013; 37:413–9.
crossref
17. Korean Academy of Rehabilitation Medicine [Internet]. Seoul, Korea: Korean Academy of Rehabilitation Medicine;2018. [cited 2020 Mar 15]. Available from: https://www.karm.or.kr/bbs/index.html?code=notice&category=&gubun=&page=7&number=14679&mode=view&keyfield=&key=.
18. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ. 1997; 314:498–502.
crossref
19. Kumar A. Is “Impact” the “Factor” that matters…? (Part I). J Indian Soc Periodontol. 2018; 22:95–6.
crossref

Fig. 1.
Pattern of publication types was compared between articles published from 1977 to 1993 and 2011 to 2019. The four types of publication were demonstrated in order, in a counter-clockwise direction.
arm-2020-44-2-158f1.tif
Fig. 2.
Significant positive linear correlation was observed between the log value of the citation (C) and Altmetric Attention Score (AAS).
arm-2020-44-2-158f2.tif
Table 1.
Sources and their weights in calculating the Altmetric Attention Score
Sources Weight
News 8
Blogs 5
Wikipedia pages 3
Policy documents (per source) 3
Patents 3
Twitter 1
Sina Weibo 1
F1000/Publons/PubPeer 1
Open Syllabus 1
Google+ 1
LinkedIn 0.5
Facebook 0.25
Q & A 0.25
Video/YouTube 0.25
Reddit/Pinterest 0.25
Table 2.
Distribution patterns according to the type of publication, specific field of rehabilitation medicine, and bibliometric variables of citation and AAS
Category Number of cases (%)
Type of publication 1,088 (100)
 Original article 779 (71.6)
 Case report 265 (24.4)
 Corrigendum 15 (1.4)
 Erratum 8 (0.7)
 Images in this issue 5 (0.5)
 Review article 4 (0.4)
 Others 40 (1.0)
Specific field of rehabilitation medicine 1,088 (100)
 Brain and neurorehabilitation 363 (33.4)
 Pain and musculoskeletal rehabilitation 263 (24.2)
 Neuromuscular rehabilitation and electrodiagnostic medicine 100 (9.2)
 Pediatric rehabilitation 96 (8.8)
 Spinal cord injury rehabilitation 91 (8.4)
 Cardiopulmonary rehabilitation 55 (5.1)
 Geriatrics and cancer rehabilitation 53 (4.9)
 Prosthetics and orthotics 20 (1.8)
 Physical medicine 16 (1.5)
 Sports rehabilitation 14 (1.3)
 Others 17 (1.6)
Bibliometrics 1,088 (100)
 No citation / No AAS 250 (23.0)
 At least 1 citation / No AAS 567 (52.1)
 No citation / At least 1 AAS 38 (3.5)
 At least 1 citation / At least 1 AAS 233 (21.4)

AAS, Altmetric Attention Score.

TOOLS
Similar articles