Scientometrics

[2][3][4][5] Modern scientometrics is mostly based on the work of Derek J. de Solla Price and Eugene Garfield. The latter created the Science Citation Index[1] and founded the Institute for Scientific Information which is heavily used for scientometric analysis. A dedicated academic journal, Scientometrics, was established in 1978. The industrialization of science increased the number of publications and research outcomes and the rise of the computers allowed effective analysis of this data.[6] While the sociology of science focused on the behavior of scientists, scientometrics focused on the analysis of publications.[1] Accordingly, scientometrics is also referred to as the scientific and empirical study of science and its outcomes.[7][8]

The International Society for Scientometrics and Informetrics founded in 1993 is an association of professionals in the field.[9]

Later, around the turn of the century, evaluation and ranking of scientists and institutions came more into the spotlights. Based on bibliometric analysis of scientific publications and citations, the Academic Ranking of World Universities ("Shanghai ranking") was first published in 2004 by the Shanghai Jiao Tong University. Impact factors became an important tool to choose between different journals. Rankings such as the Academic Ranking of World Universities and the Times Higher Education World University Rankings (THE-ranking) became an indicator for the status of universities. The h-index became an important indicator of the productivity and impact of the work of a scientist. However, alternative author-level metrics have been proposed.[10][11]

Around the same time, the interest of governments in evaluating research for the purpose of assessing the impact of science funding increased. As the investments in scientific research were included as part of the U.S. American Recovery and Reinvestment Act of 2009 (ARRA), a major economic stimulus package, programs like STAR METRICS were set up to assess if the positive impact on the economy would actually occur.[12]

Methods of research include qualitative, quantitative and computational approaches. The main focus of studies have been on institutional productivity comparisons, institutional research rankings, journal rankings[7][8][13] establishing faculty productivity and tenure standards,[14] assessing the influence of top scholarly articles,[15] and developing profiles of top authors and institutions in terms of research performance.[16]

One significant finding in the field is a principle of cost escalation to the effect that achieving further findings at a given level of importance grow exponentially more costly in the expenditure of effort and resources. However, new algorithmic methods in search, machine learning and data mining are showing that is not the case for many information retrieval and extraction-based problems.

More recent methods rely on open source and open data to ensure transparency and reproducibility in line with modern open science requirements. For instance, the Unpaywall index and attendant research on open access trends is based on data retrieved from OAI-PMH endpoints of thousands of open archives provided by libraries and institutions worldwide.[17]

Recommendations to avoid common errors in scientometrics include: select topics with sufficient data; use data mining and web scraping, combine methods, and eliminate "false positives".[18][19] It is also necessary to understand the limits of search engines (e.g. Web of Science, Scopus and Google Scholar) which fail to index thousands of studies in small journals and underdeveloped countries.[20]

Indexes may be classified as article-level metrics, author-level metrics, and journal-level metrics depending on which feature they evaluate.

In scholarly and scientific publishing, altmetrics are non-traditional bibliometrics[31] proposed as an alternative[32] or complement[33] to more traditional citation impact metrics, such as impact factor and h-index.[34] The term altmetrics was proposed in 2010,[35] as a generalization of article level metrics,[36] and has its roots in the #altmetrics hashtag. Although altmetrics are often thought of as metrics about articles, they can be applied to people, journals, books, data sets, presentations, videos, source code repositories, web pages, etc. Altmetrics use public APIs across platforms to gather data with open scripts and algorithms. Altmetrics did not originally cover citation counts,[37] but calculate scholar impact based on diverse online research output, such as social media, online news media, online reference managers and so on.[38][39] It demonstrates both the impact and the detailed composition of the impact.[35] Altmetrics could be applied to research filter,[35] promotion and tenure dossiers, grant applications[40][41] and for ranking newly-published articles in academic search engines.[42]

Critics have argued that over-reliance on scientometrics has created a system of perverse incentives, producing a publish or perish environment that leads to low quality research.[43]

• Harnad, S. (2009). "Open Access Scientometrics and the UK Research Assessment Exercise". Scientometrics. 79 (1): 147–156. arXiv:cs/0703131. CiteSeerX 10.1.1.561.7204. doi:10.1007/s11192-009-0409-z. S2CID 3183215.
• Harnad, S (2008). "Validating Research Performance Metrics Against Peer Rankings". Ethics in Science and Environmental Politics. 8 (11): 103–107. doi:10.3354/esep00088.
• The Places & Spaces: Mapping Science exhibit at the American Museum of Science and Energy, September 7, 2007 – January 7, 2008.
• Over-optimization of academic publishing metrics: observing Goodhart's Law in action GigaScience, Volume 8, Issue 6, June 2019