PDF
Citation
Print
INTRODUCTION
In the era of digitization of scholarly publishing, opportunities for improving visibility of researchers and authors from any corner of the world are increasing rapidly. The indexing and permanent archiving of published journal articles and other scholarly items with assigned identifiers, such as Digital Object Identifier (DOI) from CrossRef, help create online profiles, which play an important role in the evaluation of research performance throughout an academic career.12 Researchers need such profiles to show off their academic accomplishments, interact with potential collaborators, and successfully compete for funding and promotion.
Choosing ethical, widely visible, and professionally relevant sources for publishing research data is the first step toward making a global impact by skilled researchers. The choice of target journals is critical in the times of proliferation of bogus journals, which increase their number of articles at the expense of the quality, visibility, and citability. Researchers should be trained to manage their online profiles by listing the most valuable and widely visible works, which may attract attention of both professionals and the public-at-large. It is equally important to understand the value and relevance of the currently available evaluation metrics, which can be displayed on individual profiles by sourcing information from various bibliographic databases, search engines, and social networking platforms.
This article aims to overview current approaches to comprehensive research evaluation and provide recommendations on optimal use of widely employed author-level metrics.
AUTHOR PROFILES IN THE CONTEXT OF RESEARCH EVALUATION
Scopus and Web of Science are currently the most prestigious multidisciplinary bibliographic databases, which provide information for comprehensive evaluation of research productivity across numerous academic disciplines. Notably, Scopus is the largest multidisciplinary database, which indexes not only English, but also numerous non-English scholarly sources.3 Professionals can find a critical number of Scopus-indexed sources, reflecting scientific progress and impact in their subject categories. The database has adopted stringent indexing and re-evaluation strategies to maintain the list of ethically sound and influential journals and delist sources that fail to meet publication ethics and bibliometric standards.4 Unsurprisingly, publication and citation data in the most prestigious global academic ranking systems, such as the Times Higher Education World University Rankings, QS World University Rankings, and Shanghai Rankings, are currently sourced from the Scopus database. Scopus author profiles can be viewed as highly informative for most non-Anglophone countries and emerging scientific powers. Mistakes in the author profiles and availability of several automatically generated identifiers for the same authors can be easily corrected by pointing to the inaccuracies and requesting profile mergers at the Scopus customer service domain.
Web of Science and Google Scholar author profiles are also processed for author impact evaluations, though emerging evidence from comparative analyses does not favor these platforms on calculation of citations and feasibility grounds.567 Compared with Scopus, Science Citation Index, and Social Science Citation Index, the most selective databases of Web of Science, index much less journals, which are predominantly in English, thereby disadvantaging profiles of non-Anglophone researchers and authors, who are better represented at Scopus, Google Scholar, and regional/local indexing platforms. Google Scholar profiles often contain information about multilingual publications, which are not covered by prestigious citation databases. Such profiles are valuable for countries with limited coverage of their journals by Scopus and Web of Science (e.g., Russia, Kazakhstan, other Central Asian countries) and for authors of online monographs, book chapters, and non-peer-reviewed and grey literature items.8
Google Scholar provides information about citations of authors by tracking a wide variety of online journals, book chapters, conference papers, web pages, and grey literature items. However, several versions of the same items may appear in Google searches, affecting the automatic calculation of citation metrics. Also, this search engine fails to recognize and exclude from bibliometric calculations low-quality and apparently ‘predatory’ sources.9 The authors themselves may intentionally or unintentionally fill in their Google Scholar profiles with substandard items and articles of other authors with identical names. These and many other caveats are highlighted by experts, who disqualify author impact metrics generated by this platform.10
Regional indexing services, such as the Russian Science Citation Index (RSCI), Chinese Science Citation Database, and Indian Citation Index, fill gaps in research evaluation of authors underrepresented by Scopus and Web of Science. In fact, of 9,560 journals currently published in Russia, only 1.8% are indexed by Web of Science and 3.5% by Scopus.11 Moreover, the proportion of Russian articles tracked by the Journal Citation Reports' Science Edition (Thomson Reuters) is extremely small (1.7% in 2010).12 To improve the citation tracking of Russian academic journals, the RSCI was launched in 2009, providing an opportunity for a sizable proportion of Eurasian authors to register, acquire profiles, and monitor their locally generated metrics. With more than 650 Russian core journals selected for coverage by the Web of Science platform in 2016,13 Eurasian authors, and especially those of disadvantaged subject categories (e.g., social sciences, education, humanities) improved their global visibility and prestige through locally generated citation metrics. Although the indexing criteria of RSCI and regularity of updating its contents are different from those of the global bibliographic databases, the availability of local author impact metrics can be instrumental for further research evaluations.
Additional information on author-level impact can be gathered from emerging scholarly networking platforms, such as ResearchGate and ImpactStory, combining traditional publication and citation records with indicators of social attractiveness of scholarly items (e.g., reads, online mentions).1415 ResearchGate, a social platform with its own policy of open archiving, tracking citations, networking, and sharing archived works, depends chiefly on items uploaded by its users, some of whom are not so active in doing so, limiting the reliability of related author metrics.16 The automatically calculated ResearchGate scores are displayed on registered user profiles and processed for ranking by the platform. Although the ResearchGate platform has more than 14 million registrants, related profiles are unevenly distributed across academic disciplines and countries. Additionally, items uploaded by its users do not pass the quality checks, leaving a room for promoting substandard sources and further limiting the value of the calculated scores.
TRADITIONAL AUTHOR IMPACT METRICS
h-index
One of the widely promoted and highly cited author metrics is the h-index. It was proposed by physicist Jorge Hirsch in 2005 to quantify the cumulative impact of an individual's scientific articles, which are tracked by a citation database (Web of Science at the time of proposal).17 Jorge Hirsch suggested to analyze both publication and citation records. He defined his index as the h number of papers (Np) with at least h citations each and the rest of papers (Np−h) with equal or less than h citations each. This metric can be manually calculated by listing an individual's articles next to their citation counts in decreasing order. For example, if an author published 100 articles, each of which is cited at least 100 times, his/her h-index is 100.
By comparing the proposed index with single-number metrics, such as number of publications, citations, and citations per paper, several advantages of the combined approach were noted by J. Hirsch. He considered papers with at least h citations (“Hirsch core”) as the most impactful and important for an individual's achievements in his/her professional area.18 An important conclusion drawn by J. Hirsch from an analysis of h-indices of Nobel laureates in physics was that a breakthrough in current science is possible with continuing scientific effort over a certain period of time and other authors' positive attitude, which is reflected in increasing citations of innovative research items.
Since its proposal in 2005, the h-index has been endorsed globally as a simple, intuitive, and universally applicable metric, which can be automatically calculated and displayed on author profiles at Scopus, Web of Science, and Google Scholar. Remarkably, it has revolutionized research evaluation strategies.19
As of February10, 2018, J. Hirsch's index article is cited 3,533 times by Scopus-index items (Fig. 1). The growing global interest to the issue of the h-index and related evaluation metrics is reflected in the snapshot Scopus-based analysis of the 3,533 citing items. Of these items, 84 are cited at least 84 times (h-index of this research topic = 84). The citation activity rapidly increased over the period of 2005–2017, and reached a peak with 411 citations in 2013. The number of citations for 2018 is relatively small, but it will most probably increase with complete processing of the annual records.
Fig. 1
Number of Scopus-indexed items citing J. Hirsch's landmark article on the h-index in 2005–2018 (as of February 10, 2018).
The top 5 sources of the citations are Scientometrics (n = 415), J Informetr (n = 227), J Am Soc Inf Sci Technol (n = 97), PLOS One (n = 83), and Lecture Notes in Computer Science (n = 61). The top 5 citing authors are leading experts in scientometrics Ronald Rousseau (Leuven, Belgium, number of publications with the index citation = 54), Leo Egghe (Diepenbeek, Belgium, n = 47), Lutz Bornmann (Munich, Germany, n = 46), Jean A. Eloy (Newark, United States, n = 35), and Peter F. Svider (Detroit, United States, n = 33). The USA is the leading country in terms of citation activity with 864 related documents, followed by China (n = 348), Spain (n = 272), the United Kingdom (n = 259), and Germany (n = 234). The largest proportion of citing documents are articles (n = 2,456, 69.5%), followed by conference papers (n = 452, 12.8%), and reviews (n = 239, 6.8%). The main subject areas of the citing items are computer science (n = 1,499, 42.4%), social sciences (n = 1,137, 32.2%), and medicine (n = 761, 21.5%).
Numerous studies have employed the h-index for evaluating the lifetime performance of scholars across most fields of science. This metric is widely employed for academic promotion, research fund allocation, and evaluation of the journal editorial board prestige.202122 The results of several studies have allowed extending its use for ranking journals, research topics, institutions, and countries.23242526
Despite some advantages for evaluating influential, or ‘elite,’ researchers, the h-index has been criticized for numerous limitations and inappropriate uses.27 Even though related thresholds are often arbitrary, this index is apparently inappropriate for ranking authors with a few publications and citations. At the same time, scores of the index for ‘elite’ researchers can increase with growing citations to their ‘old’ articles despite decreasing their overall publication activity, thus disqualifying it as a metric of dynamic changes in productivity. With the digitization and expanded indexing of historic papers, the h-index scores of deceased authors can increase, giving a false impression of growing productivity.24
The h-index calculations are insensitive to article types, highly cited items, total citations, self-citations, public accessibility (open access), number and gender of co-authors, cooperation networks, and referencing patterns, research funding, differing widely across disciplines and countries. On the dark side, some authors, and particularly those with a small number of publications and citations, may be tempted to intentionally inflate their scores by irrelevant self-citations, resulting in annual increases of the h-index by at least 1. Such manipulations can be easily detected by calculation of a related score (q-index) and contextual qualitative analyses of citation patterns.28
A recent large bibliometric analysis of 935 US-based plastic surgeons revealed that research-specific funding and higher funding amounts significantly correlate with their h-index scores.29
The importance and value of articles indexed by specialist databases are not reflected in the h-indices. For example, the list of items indexed by the evidence-based MEDLINE database is more important for biomedical authors than their publication records in other databases.30
The h-index values of the same author vary widely, depending on the employed citation-tacking platform.31 Compared with Scopus and Web of Science, Google Scholar calculations often provide higher values of the index. Any such calculations, however, overlook the scientific prestige, ethical soundness (predatory and retracted vs. ethical and actively indexed items), and context of publications and citations. The articles in top journals increase scientific prestige and academic competitiveness of their authors. The same is true for citations from top journals.
h-index variants
At least 50 variants of the h-index have been proposed to overcome some of its disadvantages and improve numerical analyses of an individual's research productivity.32 Nonetheless, an analysis of 37 variants revealed that most of the proposed alternative metrics correlate highly with the original h-index and add no new dimension for measuring research productivity.33 And above all, all new indicators have a number of specific limitations (Table 1), which point to the need of their further validation by experts in bibliometrics.
Table 1
Main strengths and limitations of some author-level metrics
To visualize publication and citation dynamics throughout an individual's career, the timed h-index was proposed.34 It is calculated in analogy with the original h-index, but with restriction for a time window of 5 years in Web of Science. Such an approach focuses on recent publications and citations rather than lifetime performance and compares activity at various 5-year time points. Similar empirical approaches have been experimented by restricting time window to 1 and 3 years.3536 The shorter time window, the lower values of the resultant h-index scores. These indexes can be calculated for various periods of time, depending on specifics of publication and citation traditions across academic disciplines.
A 5-year time window was considered for the Author Impact Factor (AIF), which is calculated in analogy with the Journal Impact Factor.37 The AIF is the number of citations to an author's articles, which are published in a certain year, divided by the number of the evaluated author's articles in the previous 5-year period (based on data from Web of Science). Like the timed h-index, the AIF is sensitive to fluctuations in publication and citation activity over certain periods and may reveal an unusual increase of citations to an author's work(s) following publication of groundbreaking papers.
Also, J. Hirsch proposed the m-quotient to correct the h-index for career length and compare the impact of researchers with various periods of academic activities in the same field. The m-quotient is calculated by dividing the h-index score by the number of years since the evaluated author's first publication.17 This metric correlates positively with academic rank of evaluated individuals.38 Like the h-index, m-quotient is unsuitable for evaluating early career researchers. Besides, the first publication is not always the start of an active career in a certain field. It takes years until a researcher finds his/her academic niche. Finally, the m-quotient overlooks interruptions in an individual's career, which can be an important issue particularly for young and female researchers. To overcome this limitation, the contemporary h-index (hc) was proposed to consider age of each article, give more weight to recent articles, and credit researchers with lasting contributions to a specific field of science.39
g-index
In 2006, Leo Egghe40 designed another metric to focus on a set of highly-cited articles. His proposal has received 802 citations in Scopus (as of February 10, 2018). The g-index can be calculated on the Harzing's Publish or Perish website, using data from Google Scholar or subscription citation databases.41 To compute this index, a set of articles is ranked in decreasing order of the number of citations. The resultant score is the largest number of top g articles cited at least g2 times. Roughly, the h-index requires a reasonable number of cited articles to get a high score whereas the g-index can be high even with a few highly cited articles.42 For the same individual, the g-index is always greater than the h-index. For example, an author with 10 published articles, three of which are cited 60, 30, and 10 times (100 in total), will get g-index of 10 and h-index of 3.
e-index
Chun-Ting Zhang43 proposed the e-index to rank researchers with identical h-index scores but different total citations. As of February 10, 2018, the article is cited 170 times in Scopus. This complementary index helps distinguish researchers with excessive total citations. It is an informative indicator for highly cited researchers, and particularly for Nobel laureates. Although the g-index is sensitive to highly cited items, the e-index is a more accurate metric for evaluating differences in the Hirsch core citations of highly cited researchers.44
n-index
There is a field-normalization approach to evaluate researchers with identical h-index scores. The Namazi (n)-index was proposed to compare researchers' impact in view of citation patterns in related scientific fields.45 This index is calculated by dividing an individual's h-index by h-index of a top journal in a related subject category. Information is sourced from Scopus and SCImago Journal & Country Rank platform. Despite its originality, the n-index has not attracted much attention, primarily because of the lack of mathematic justification of the idea. In practical terms, the resultant n-index scores are small numbers. The n-index can be calculated using Scopus data only. Also, the employed individual and journal metrics reflect different phenomena, making the proposed formula difficult to justify.
PageRank index
A completely different approach to the evaluation of an individual's impact by correcting for possible citation manipulations is undertaken by the originators of the PageRank index.4647 The proposal is still new and not widely endorsed. The idea refers to the analysis of citation networks using the PageRank algorithm of Google. It gives more weight to citations from webpages with high PageRank score and allows distinguishing early career researchers with even a few innovative publications, attracting attention of citing authors. Essentially, the PageRank index does not consider the quantity as does the h-index, it pays more attention to innovative dimension and the ‘quality’ of the analyzed publications. The computation of the PageRank index is sophisticated and not readily understandable to non-experts. What is easy to understand is that the first step toward increasing the impact is to gain more visibility on Google Scholar for a set of articles, which may increase chances of citations from widely visible, highly prestigious, and well cited online items. The index can increase even with a few citations from articles which attract a growing number of citations.
Total publications
One of the main limitations of the h-index relates to its focus on a large number of publications as a reflection of productivity, which is achievable over a certain period of academic activities, and particularly in a team of experts of prestigious institutions, who work on ‘hot’ research topics with proper funding, supply of modern equipments, and support of co-authors and collaborators.4849
As a notable example, leading American cardiologists, who are involved in cardiovascular drug trials at prestigious research centers, have astronomically high h-index scores (e.g., Eugene B. Braunwald and Paul M. Ridker have 1,100 and 959 Scopus-indexed articles with 182 and 177 h-index scores, respectively). Apparently, the h-index cannot be employed for ranking and promotion of early career researchers, academic faculty members overburdened with teaching responsibilities, or experts of emerging disciplines with a small number of articles.
To increase their h-index scores, researchers have to publish more in the first place and increase their total publications, which is the basic activity indicator.50 Scopus-based analyses, for example, generate graphs with details of who publishes most in certain fields. This type of information can be employed for mapping the scientific progress.51
While high publication counts can be associated with great achievements, innovations, and high citation counts in developed countries,5253 sloppy and otherwise unethical publications may occasionally inflate this measure of productivity elsewhere. Some researchers erroneously believe that the higher the number of publications, the greater the chances of making an impact and achieving a high academic rank. Such beliefs, which are dominant in some non-mainstream science countries, have led to the growth of unjustified, ‘salami-sliced,’ inconclusive, redundant, and pseudoscientific publications.545556 Alarmingly, the unprecedented growth of secondary publications, such as systematic reviews and meta-analyses, often generated on common templates, pollutes the evidence base in oncology, medical genetics, and other critically important fields of clinical medicine and public health.5758 By comparing records of active and non-active researchers, the evaluators should recognize and filter out any ‘useless’ and nonsense publications.
Total publication count as a quantitative metric overlooks the relevance of article type, number of co-authors, subject category, indexing database, and value of publishing in both national and top journals (e.g., Nature, Science). Publication records of early career researchers can be easily inflated by numerous case studies, which rarely find their way to top journals, weakly influence the evidence accumulation and bibliometric analyses, and often convey a practical message only.596061 Publication counts are also high among authors who produce numerous letters and commentaries, dispersed across journals of various academic disciplines. Such practice of boosting numbers by generating pointless items is harshly criticized as counterproductive and unethical.55
Although the authors themselves can collect their publications from various indexed and non-indexed sources and display related records on their online profiles,2 evaluators have to navigate through several platforms, including related national citation indexes and specialist databases, to comprehensively and accurately judge the publication outputs. Korean research managers, for example, prioritize not just total publications, but articles indexed by the Korean Citation Index, which attract more local citations than those indexed by global databases and remained unnoticed for the global community.62
Busy practitioners, academic faculty members, and researchers in emerging fields of science usually produce much less and lag behind their colleagues, who work in established research areas. Evidence suggests that elementary mentorship and writing support by peers and professional writers, which is aimed at adhering to available standards and citing reliable sources, effectively and ethically increases publication activity.636465
Total citations
Citation counts reflect readers and citing authors' interest to a certain set of articles. To some extent, this indicator is a surrogate measure of the quality of research. Authors with several thousand citations, received in the course of long-lasting academic activities, can be viewed as the generators of quality items, influencing the scientific progress.
In medicine, large amount of citations in an individual's profile is often due to authoring innovative methodology reports, clinical trials, large cohort studies, systematic reviews, and practice recommendations.666768 A notable example is the case of Oliver H. Lowry,69 who published his paper on protein measurement with the folin phenol reagent in 1951. As of February 10, 2018, Lowry's profile in Scopus records 261,840 total citations for 226 documents with 244,589 linked to the seminal methodology paper.
Evidence from a systematic review suggests that authors who publish statistically significant results, supporting their working hypotheses, receive 1.6–2.7 times more citations than those who, quite correctly, publish non-significant and negative results.70
The detailed analysis of the distribution of citations provides crucial information about the context of research productivity. An individual's single landmark article may garner more citations than all other publications altogether.7172 The excessive citations to a single article are often eyeballing, with Oliver H. Lowry's methodology paper being the best example. The citation speed with which published articles attract citing authors' attention can be also eyeballing and reveal researchers with truly innovative and popular research studies.73
A recent bibliometric study suggested that the growing citation counts in the past 5 years, reflected in the h5-index scores, can reveal candidates with a growing influence and the best chances for academic promotion.74
A larger time window is employed by Clarivate Analytics to distinguish exceptional and actively publishing researchers. Researchers are ranked as highly cited if their articles are regularly listed in the top first percentile of highly cited items in their subject category of Web of Science over the past 10 years. Since 2014, more than 3,000 highly cited researchers have been grouped in one of 21 subject fields and credited annually by Clarivate Analytics.75 The results of related annual reports of the Essential Science Indicators (Web of Science component) have influenced the world's university rankings by the Academic Ranking of World Universities.76
Despite the growing role of citation counts for crediting previously published works,77 it is also increasingly important to identify irrelevant auto- and reciprocal, or ‘friendly,’ citations that artificially inflate an individual's profile.78 These unethical citations, which often originate from non-mainstream science countries and substandard sources, primarily skew the evaluation of authors with a small number of citations.79
EMERGING ALTERNATIVE METRICS
Citation counts and related traditional metrics are dependent on the authors' citing behavior and speed with which their subsequent articles are published. Even in the times of the wide availability of fast-track online periodicals, the publication process is still lengthy, which makes it impossible to evaluate immediate public attention to research output. The Altmetrics indicators, which are displayed on Altmetrics.com (supported by Macmillan Science and Education, the owner of Nature Publishing Group), reflect public attention in a wider context and counterbalance some of the limitations of the citation metrics. The website was established in 2011. Downloads, reads, bookmarks, shares, mentions, and comments on published items at any, not necessarily scholarly, outlets of social media are now available to analyze public value, or “societal impact,” of publications.8081 The detailed analyses may help ascertain geography and professional interests of the immediate evaluators, who tweet, share, and positively mark relevant and valuable items on social media channels. The evaluators are not just researchers and potential authors, but also trainees, practitioners, and public figures.82 Therefore, careful interpretation of the context of social media attention is advisable.83
Scholarly users of social media are unevenly distributed across disciplines, with social scientists representing the most active and life scientists the least active groups.84 Such discrepancies necessitate field normalization.
One of the scholarly components of the alternative impact evaluation takes into account comments and recommendations posted on Faculty of 1,000 Prime, PubPeer, Publons, and other emerging services for publicizing peer review and crediting peer reviewers.85 A recent analysis of research performance at Google Scholar and peer-review activities of scholars registered with Publons revealed that the peer-review forum provides scholarly information of great importance to journal editors.86 Publons has been also recognized as a supplier of data for quantifying reviewer activities and combining verifiable alternative metrics with traditional ones.87
Although no author-level composite alternative metric is currently available, the Altmetric Attention Scores of separate items can be grouped to distinguish the most attractive articles at an individual's online profile. These scores for items with digital identifiers, such as DOI and PubMed ID, can be freely computed using the Altmetric bookmarklet tool.88
CONCLUSION
The evaluation of individual research productivity is a complex procedure that takes into account volume of research output and its influence (Table 2). Apparently, no single metric is capable of comprehensively reflecting a scope of scholarly activities. Concurrently, a single scholarly work with innovative design and outcomes may attract global attention, point to a great scholarly achievement, and necessitate crediting. Readers and prospective authors are the main evaluators, who refer to published works on their social networking platforms and scholarly articles, highlighting strengths and limitations of research data. Apparently, those who wish to increase their scholarly impact should be concerned not just with volume, but also with the quality, visibility, and openness of their works.89 Carefully editing, structuring, and adequately illustrating articles may increase the chances of publishing in widely visible and highly prestigious sources. Visibility of research may increase attention of readers and attract constructive post-publication criticism along with relevant citations.
Table 2
Pointers for employing researcher and author impact metrics
Tools for quantitative evaluation of the researcher and author impact have proliferated over the past decade. This trend is mainly due to the proposal of the h-index in 2005 and numerous attempts to overcome its limitations.90 The h-index as a simple and easily understandable metric has been well accepted by evaluators. Several variants of the h-index have been proposed to correct for some, but not all confounders (e.g., age of academic activities, level of activity in the past 5 years, excessive citations to some items). However, the initial enthusiasm toward the simplicity of the quantitative evaluation has been overshadowed by concerns over the inappropriate uses of individual metrics and prompted more detailed analyses of publication and citation sets. Professional relevance of publications and context of citations often reveal the true impact of individual academic activities. The analysis of citation sources and their scientific prestige provide more valuable information than simply recording citation counts.
Research productivity includes not just scholarly articles, but also peer reviewer comments, which can be counted as units of publication activity provided these are properly digitized and credited by evaluators. The Publons platform (Clarivate Analytics) is currently the emerging hub for publicizing reviewer comments and ranking skilled contributors to the journal quality. Registering with Publons and transferring information to individual online identifiers is an opportunity to further diversify researcher and author impact assessment.
Prospective use of researcher and author metrics is increasingly dependent on evaluation purposes. The evaluators should take a balanced approach and consider information sourced from various global, national, and specialist bibliographic databases.
XML Download