Bibliometrics is a form of research that combines the traditional bibliographic research of analyzing the information obtained from published materials with a technical method to quantitatively calculate the impact of the publications [1]. Recently, the number of global reports created by individuals in the medical field utilizing bibliometrics has increased [2].

Since the concept of the Impact Factor (IF), which quantifies the degree of citations within an expert group was first introduced in 1975, the indices for journals listed in the Journal Citation Reports (JCR) have been calculated annually. The IF is now the most objective indicator when referring to the conventional metrics (CM) of bibliometrics [3]. However, with the development of innovative mobile devices and platforms, there is a rapidly growing demand for real-time quantitative measurements for the public dissemination of scientific knowledge through various media. In response to this real-time public demand, using alternative metrics (AM) has been proposed as a new research methodology and as an alternative tool to complement CM [4].

Altmetric LLP, a leading AM company based in London, England [5], has designed specific algorithms to evaluate the public dissemination influence of a scientific article through social media. Using these algorithms, they have developed an Altmetric Attention Score (AAS) for scientific articles through their website (https://www.altmetric.com). The publication of research using the AAS has increased in a variety of medical sciences [6–8].

In particular, as rehabilitation medicine manages both acute and chronic therapeutic care, a multidisciplinary treatment approach that encourages patients and their families to participate in the long-term recovery phase is essential. It requires the rehabilitation medicine necessary for synchronizing insights between medical professional groups and the public. Nevertheless, there have only been a few studies that have investigated international rehabilitation-related journals using alternative bibliometrics [9–11].

The Korean Academy of Rehabilitation Medicine, the representative academic society for rehabilitation medicine in the Republic of Korea, began publishing an official journal in 1977 and has regularly published it six times a year based on an English-only policy since 2011 [12]. The Annals of Rehabilitation Medicine (ARM) is the name of this journal and several bibliometric studies have been published in it with respect to overall compositions [13], statistical methods [14], research design [15], publication ratio of abstracts presented at the annual meeting of the society [16], and the status of ARM at the international level [12]. However, there have been no studies using AM methodologies in ARM.

This study aims to investigate ARM articles using a bibliometric analysis that includes conventional and AM and to verify whether there is a correlation between the topics of interest between expert groups and the public media.

The ARM website (https://www.e-arm.org) lists the articles published in each issue and offers the ability to search these articles through open-access. To analyze the bibliometrics that correlate with international relevance, we included only ARM articles published since the third issue of 2011, when the society introduced an English-only policy.

This study simultaneously adopted both CM and AM methodologies for the bibliometric analysis of ARM. The CM was investigated by analyzing the type of publication, number of citations, and the specific field of rehabilitation medicine for each article. The types of publications were divided into four categories: review articles, original articles, case reports, and others. The number of citations per article was verified by using CrossRef’s text services, which are provided in the banner on the ARM website. The specific fields were distinguished as ten subjects, depending on the way in which the academy categorized the affiliated members who were fourth-year resident doctors in the annual examination for the specialized rehabilitation medicine qualifications. The subjects are as follows: “brain and neurorehabilitation”, “neuromuscular rehabilitation and electrodiagnostic medicine”, “pain and musculoskeletal rehabilitation”, “spinal cord injury rehabilitation”, “physical medicine”, “pediatric rehabilitation”, “prosthetics and orthotics”, “geriatrics and cancer rehabilitation”, “cardiopulmonary rehabilitation”, and “sports rehabilitation” [17].

The AM was analyzed using the data provided by Altmetric; their website provides the AAS, which is a weighted score of the total mentions of the article across various online media (Table 1). The score is also provided on the banner of the Altmetric website; however, it is limited to confirming the specific details of the score. Altmetric formally approved this study, and so detailed data could be obtained and analyzed without restriction. The data were collected on a specific day (June 30, 2019) to avoid changes in the online activities of the articles.

Two specialists in rehabilitation medicine, who have both worked at a general hospital for more than 10 years as a trainer for resident doctors independently, conducted a review and analysis for the articles. If these two reviewers disagreed, a consensus was achieved through open discussion. The local ethics committee reviewed and approved an exemption for the study.

All of the statistical analyses were performed using IBM’s SPSS version 19.0 for Windows (IBM, Armonk, NY, USA). The distribution patterns according to the type of ARM publications were compared with previous data and measured at different publication times for the journal using a chi-squared test. The Spearman rank correlation coefficient was used to evaluate the correlation between the number of citations and the AAS; for all the tests, the statistical significance was set at p<0.05.

From the third issue of 2011 to the third issue of 2019, ARM published 1,088 articles, and the distribution pattern for the bibliographic variables of ARM was identified (Table 2). First, with respect to the type of publication, there were 779 original articles, which was the highest percentage (71.6%), and this was followed by the percentage of case reports (24.4%), others (1.0%), and review articles (0.4%) in descending order. From a study of 559 ARM articles from the previous version of the journal that were published from 1977 to 1993, Lee et al. [13] reported that the most common type of publication type was original articles, followed by case reports, review articles, and others in descending order. Comparing the results of that study with this study, there was a significant difference in the distribution pattern of the type of publication (p<0.05) (Fig. 1).

Second, for the specific field of rehabilitation medicine, the number (percentage) of articles regarding brain and neuro-rehabilitation and pain and musculoskeletal rehabilitation was 363 (33.4%) and 263 (24.2%), respectively. The combination of the two fields accounted for more than 50% of all of the articles and they were followed by articles on neuromuscular rehabilitation and electrodiagnostic medicine, pediatric rehabilitation, spinal cord injury rehabilitation, cardiopulmonary rehabilitation, geriatrics and cancer rehabilitation, prosthetics and orthotics, physical medicine, and sports rehabilitation in descending order.

In the bibliometric analysis, the number of articles cited by a journal at least once was 800, and the citations were in the range of 1 to 128 (mean, 5.60±6.22). The number of articles with at least one AAS point was 271, and the AAS was in the range of 1 to 127 (mean, 3.20±8.92) (Table 2). The detailed data that constitutes the AAS were obtained through the Altmetric website, and for all of the 271 articles included there were 1,003 mentions on Twitter (67%), Facebook (14.3%), and Sina Weibo (6.88%), and in patents (6.78%) and the news (2.39%).

There were 233 articles satisfying both conditions of at least one citation and at least one AAS point (Table 2). For those articles, as each citation distribution and AAS deviated to the lower values, the scores were converted to log values for the parametric statistical analysis. The number of citations and the AAS were found to have a statistically significant positive linear correlation on the scatter plot (r=0.216, p=0.001) (Fig. 2).

This study was conducted to investigate the ARM articles using a bibliometric analysis involving both conventional and AM. It demonstrated that there was a significant correlation between the citations in academic journals and the mentions on public media.

Since the concept of bibliometrics was introduced into the field of medicine, the IF for CM has been essential for discussions about the journal. Most researchers strive to publish their articles in a journal with a higher IF, and it is common for schools, institutions, and hospitals to use this as an objective criterion to judge performance [3]. However, as the IF is updated only once a year and has a large variation in each academic field, it has limitations. In addition, when the IF is emphasized too much, there could be a distorted phenomenon, and the composition and editing direction of the journal can adjust to it [18]. As a complementary concept, AM has been discussed actively to measure the dissemination of scientific knowledge to a public audience through social media. To evaluate a particular article’s influence, AAS has been created by Altmetric, which is a weighted score of the total mentions of the article across various online media (Table 1). AAS is based on three main domains: authors, sources, and volume. To rate a score, they gather information from diverse online media sources. After putting it all together, the AAS indicates a quantified approximation of all the real-time attention picked up for a research output. The score is useful when looking at several articles together in order to identify the level of online activity related with a particular research output [6–8,11].

Since 1995, when the first bibliometric study was published, various bibliometrics-related research had been published by ARM. In particular, Huh [12] reported that the change of the ARM language policy to English-only was successful in elevating the status of the journal to the international level and subsequently analysis using CM became possible. In line with Huh’s suggestions for meeting international standards, the ARM website currently offers metric data provided by CrossRef. However, there have been no studies that have applied alternate metrics to the bibliometric analysis of ARM articles.

This study investigated the bibliometric analysis using both CM and AM methodologies. First, with respect to the CM, the ratio of original articles was the highest, followed by case reports. The combined ratio of both was over 90%, and this was similar to the results from a previous study [13]. However, when this was examined in detail, original articles were observed to be decreasing, and the ratio of review articles had also reduced remarkably— and this was statistically significant. Considering the findings that more review articles lead to more citations and a higher IF, it is necessary to encourage the submission and publication of review articles [19].

From the CM, for the 10 specific fields of rehabilitation medicine, the ratio of brain and neuro-rehabilitation and pain and musculoskeletal rehabilitation accounted for more than half of the total. Considering that the 10 subjects were selected to assess the minimal qualification by sector of resident doctors completing rehabilitation medicine courses, and the importance of diversity in the field of rehabilitation medicine, there might be a need to review the cause for this biased phenomenon by subject.

For the analysis of the AM of the ARM articles, using the data officially provided by Altmetric, the rate of Twitter mentions was overwhelmingly higher among all of the 14 types of media used in the algorithm for the calculation of AAS. This means that Twitter is the most used form of social media, and from the alternative bibliometric analysis of the journal, its influence in the spread of medical knowledge is increasing. In a study on the AM of the Journal of Physical Medicine and Rehabilitation [10], the authors reported that a social media strategy using Twitter was effective for disseminating their research and creating a larger social discourse. Since 2012, the Korean Academy of Rehabilitation Medicine has had both Twitter (https://twitter.com/karm_1972), and Facebook (https://www.facebook.com/karm.or.kr) accounts. However, they have not been actively operating them to date, and their Twitter account has only two posts and 46 followers (verified on June 30, 2019).

The most important point of this study was the comparison between the AM and the citations of the CM, and there was a significant correlation between the two variables. As in previous studies highlighting the importance of the complementary role of the AM to the CM [7,8,10], our findings imply that the effort to increase the AM could elevate the status of ARM. Therefore, a strategic approach is necessary to carry out alternative bibliometric monitoring continuously on the research achievements of the journal, and to spread the journal’s articles to the public effectively by actively utilizing social media. The view of this study is in contrast to some studies that did not reveal significant correlations between the two, which seems to be caused by a research design or period that does not consider such a comparison [9,11].

The present study was limited by an inherent shortcoming of Altmetric and therefore the validity of the social media mentions was hindered due to the anonymity of online communication. In addition, it was difficult to compare the results of this research with those of other international rehabilitation-related journals that have been granted an IF. In the future, when ARM is also given an IF, it is expected that it would be possible to analyze its bibliometrics and compare these objectively with other journals.

In conclusion, our study demonstrated that for this journal, the AM significantly correlated with the CM, and it is important to actively use social media to increase the AM. It is also important for the academic society to increase the dissemination of academic knowledge through public media, and increase the citation-related index for the status of the journal.