MESUR: implications of usage-based evaluations of scholarly status for open repositories

Abstract

The MESUR project at the Digital Library Research and Prototyping team of the Los Alamos National Laboratory's was funded by the Andrew W. Mellon Foundation to investigate the possibilities of usage-based scholarly evaluation metrics. MESUR's investigation is conducted on the basis of a large-scale reference data set that combines usage, citation and bibliographic data obtained from a wide variety of publishers, aggregators and institutions. Scholarly evaluation is at this point mostly conducted on the basis of metrics based on citation statistics, e.g. article citation counts, the Impact Factor and the h-index. Although citation data, possibly due to its origin in the published, peer-reviewed literature, has acquired an aura of validity and reliability, a number of shortcomings have been repeatedly identified that affect open repositories. Publication delays can cause citation data to lag scholarly developments by significant periods of time leading to an under appreciation of services that shorten the scholarly communication cycle. Most importantly, citation data frequently ignores the growing body of grey literature or non textual scholarly objects that exist outside the realm of scholarly journals and can thus not assess the growing impact of scholars that publish their research outside the limited set of journals tracked by Thomson Scientific or any particular commercial publisher collections. It has become apparent from the evolving MESUR research that usage data, in particular when aggregated across a multitude of scholarly information sources, is a promising complement to traditional citation data. This is particularly true for open repositories.