Leiden Manifesto

The Leiden Manifesto for research metrics (LM) is a list of "ten principles to guide research evaluation", published as a comment in Volume 520, Issue 7548 of Nature, on 22 April 2015. It was formulated by public policy professor Diana Hicks, scientometrics professor Paul Wouters, and their colleagues at the 19th International Conference on Science and Technology Indicators, held between 3–5 September 2014 in Leiden, The Netherlands.

The LM was proposed as a guide to combat misuse of bibliometrics when evaluating scientific research literature. Examples of commonly used bibliometrics for science, or scientometrics, are the h-index, impact factor, and websites displaying indicators such as Altmetrics. According to Hicks et al., these metrics often pervasively misguide evaluations of scientific material.

The Leiden Manifesto for research metrics

Illustration by David Parkins
Authors	Diana Hicks, Paul Wouters, Ludo Waltman, Sarah de Rijcke, Ismael Rafols
Publisher	Nature
Publication date	22 April 2015
Website	leidenmanifesto.org

Motivation

Motivations for codification of the Leiden Manifesto arose from a growing worry that "impact-factor obsession" was leading to inadequate judgement of scientific material that should be worthy of fair evaluation. The lead author of the LM, Diana Hicks, hoped that publishing in Nature would expand the ideas already commonplace in the scientometrics sphere to the broader scientific community. Although the principles of the LM are not new to scientometricians, Hicks et al. desired to create a unification of ideas to guide future editors and reviewers of scientific literature.

DORA and other predecessors

Interest in the reform of how research assessment is conducted is an ongoing debate in the scientific community. A well known declaration intended to quell the use of the impact factor, the San Francisco Declaration on Research Assessment was published around two years prior to the manifesto. The LM broadened the ideas presented in the DORA, which has now since been signed by over 2000 organizations and over 15000 individuals.

One of the main concerns about overuse of citation-based performance indicators came from the observation that smaller research organizations and institutions may be negatively affected by their metric indices. In one public debate at the Centre for Science and Technology Studies at Leiden University, it was acknowledged that indicators which measure citations may give "more weight to publications from fields with a high expected number of citations than to publications from fields with a low expected number of citations".

Although the main focus of the LM is based on the use of scientometrics for research evaluation, in its background, Hicks et al. also explain why overuse of metrics can adversely affect the wider scholarly community, such as the position of universities in global rankings. According to Hicks et al., scientific metrics such as citation rate are used far too much for ranking the quality of universities (and thus the quality of their research output).

Journal impact factor

Further information: Impact factor § Criticism

The background for the LM describes why misusing metrics is becoming a larger problem in the scientific community. The journal impact factor, originally created by Eugene Garfield as a method for librarians to collect data to facilitate selecting journals to purchase, is now mainly used as a method of judging journal quality. This is seen by Hicks et al. as an abuse of data in order to examine research too hastily. For example, an impact factor, while a good metric to measure the size and experience of a journal, may or may not be sufficient to accurately describe the quality of its papers, and even less so for a single paper.

Content

Consisting of ten concise principles, along with a description for each, the Leiden Manifesto aims to reconstruct the way that research evaluations by academic publishers and scientific institutions are done. Its emphasis lies in detailed and close evaluation of research, rather than the excessive use of quantitative data in evaluations. It aims to promote academic excellence and fairness with thorough scrutiny, as well as remove possible perverse incentives of using scientometrics, such as judgement of academic capability and university quality.

Ten principles

The ten principles of the Leiden Manifesto are as follows:

1. Quantitative evaluation should support qualitative, expert assessment.
2. Measure performance against the research missions of the institution, group, or researcher.
3. Protect excellence in locally relevant research.
   • That is, allow research that takes place in a certain area or field to be published in corresponding local research publication, instead of prioritizing high-impact journals. Many high-impact journals are in English, which may decrease needed specificity when publishing a paper meant to study locational characteristics. As an example, in high-impact Spanish-language papers, "topics such as local labor laws" and other features designated for sociologists may be lost.
4. Keep data collection and analytical processes open, transparent, and simple.
5. Allow those evaluated to verify data and analysis.
6. Account for variation by field in publication and citation practices.
   • Peer-review and citation rate can vary wildly across differing disciplines, for example, "top-ranked journals in mathematics have impact factors of around 3; top-ranked journals in cell biology have impact factors of about 30".
7. Base assessment of individual researchers on a qualitative judgement of their portfolio.
8. Avoid misplaced concreteness and false precision.
   • According to Hicks et al., use of scientific indicators may precede strong assumptions that are not necessarily correct. For example, when looking at a specific scientist, a low citation rate may lead the investigator to assume low research quality, which is implying causation from correlation. Providing clarification, as well as multiple, robust indicators, may reduce inappropriate concreteness. False precision is possible when indicator producers, such as Clarivate (which publishes the annual Journal Citation Reports) attempt to create an exact journal impact factor (i.e. three decimal places). Hicks, et al. argue that conceptual ambiguity and random variability of citation counts make it unnecessary to distinguish indices such as journal impact factors to such a precise extent, because it can foster excessive comparison and competition between publishers.
9. Recognize the systemic effects of assessment and indicators.
10. Scrutinize indicators regularly and update them.

Reception

2016 John Ziman Award

In 2016, the Leiden Manifesto was presented with the John Ziman Award by the European Association for the Study of Science and Technology for its effort to widen scientometrics knowledge to the scientific community as a whole. EASST president Fred Steward stated that the LM "emphasizes situatedness, in terms of different cognitive domains and research missions as well as the wider socioeconomic, national and regional context". This award contributed to the solidification of the LM as a continuing public debate for the scholarly community. The award was received well by lead author Diana Hicks.

Leiden Manifesto, The Metric Tide, and DORA

The Leiden Manifesto gained popularity following its publication, mainly by the scholarly publishing community which looked to reform their practices. The LM is often cited alongside other similar publications; those being the DORA and UK based academic review The Metric Tide. In a public statement by the University of Leeds, support for committing to using the principles of the LM along with DORA and The Metric Tide was issued in order to further research excellence.

Public endorsement

LIBER, a collaboration of European research libraries issued a substantial review on the LM in 2017. It concluded as a viewpoint that the LM was a "solid foundation" on which academic libraries could base their assessment of metrics.

Elsevier, a global leader in research publishing and information analytics, announced on 14 July 2020 that it was to endorse the LM to guide its development of improved research evaluation. Elsevier stated that the principles of the manifesto were already close in nature to their 2019 CiteScore metrics, which was in summary "improved calculation methodology" for "a more robust, fair and faster indicator of research impact". This alignment further popularized the LM, and also illustrated the shift in practices of evaluation by prominent research publications.

The Loughborough University LIS-Bibliometrics committee chose to base their principles on those of the LM instead of the DORA, because according to their policy manager Elizabeth Gadd, the LM takes a "broader approach to the responsible use of all bibliometrics across a range of disciplines and settings". Stephen Curry, the chair of the DORA steering committee, commented on this statement by emphasizing that DORA was aiming to extend its "disciplinary and geographical reach". However, he still made it clear that a university should have the right to choose to follow either the DORA or the LM, or neither, as long as reasonable justification was provided.

Further applications

David Moher et al. referenced the LM in a perspective for Issues in Science and Technology that the "right questions"(i.e. research planning, timeframe, reproducibility, and results) for assessing scientists were not being asked by academic institutions. Moher et al. criticize the obsession of journal impact factors and "gaming" by investigators of scientometrics. Instead, Moher et al. advocate for usage of DORA and the LM when assessing individual scientists and research.

T. Kanchan and K. Krishan describe by a letter in Science and Engineering Ethics why the LM is "one of the best criteria" for assessing scientific research, especially considering the "rat race" for publications in the scholarly community. Kanchan and Krishan emphasize that use of the LM will lead to "progress of science and society at large".

See also

• San Francisco Declaration on Research Assessment
• Nature (journal)
• Scientometrics (journal)
• Bibliometrics
• Scientometrics
• H-index
• Impact factor
• Altmetrics