Hundreds of delegates participated in a day-long symposium on scientific evaluation metrics, held in Santa Fe New Mexico on May 10th 2011, the result of a collaboration between Elsevier and Miriam Blake, Director of the Los Alamos National Laboratory (LANL) Research Library. This symposium focused on scientific output measurements, methodologies and mapping techniques, and was globally broadcasted from the conference ballroom, where world-renowned speakers presented and discussed existing and emerging metrics used to evaluate the value and impact of research publications.

Measuring the impact and value of scientific publications is critical as governments are increasingly seeking to distribute research funds in ways that support high-quality research in strategically important fields..  Through the years, several methodologies and metrics have been developed in order to address some of the many factors that can be taken into consideration when assessing scientific output. New evaluative metrics have emerged to capitalise on the wealth of bibliometric data in analyzing citation counts, article usage, and the emergence and significance of collaborative scientific networks. In addition, ever-increasing computational power enables rigorous relative and comparative analysis of journal citations and publication relationships to be calculated and used in unique ways, including a variety of visualization solutions.

Symposium themes generated from registrants feedback and comments

New developments

The symposium offered insight into the topic of research evaluation metrics as a whole. The discussion was headed by Dr. Eugene Garfield and Dr. Henk Moed, who addressed established and emerging trends in bibliometrics research, with both stressing the necessity of using more than one method to accurately capture the impact of research publications and authors. Emerging and innovative approaches using journal and scientific networks, weighted reference analysis and article usage data were also presented. Dr. Jevin West discussed the latest developments in the EigenFactor (; Dr. Henry Small unveiled a new method for citations text mining in emerging scientific communities; Dr. Johan Bollen presented the MESUR ( project; and Dr. Kevin Boyack ( demonstrated how co-citation analysis can be used to identify emerging and established scientific competencies within institutions as well as countries. The methodological discussion was accompanied by a demonstration of visualization solutions that capture these relationships and enable a broad view of scientific trends, networks and research foci.

More than a thousand words

The power of visualizing such networks was demonstrated by Dr. Katy Börner, who headed the discussion on the variety and diversity of scientific mapping tools. Dr. Börner brought a wealth of examples through the “Places & Spaces” ( exhibition, which was displayed in the conference room and via her presentation. Maps for scientific policy and economic decision makers, along with maps for forecasting and research references, were among the examples displayed and discussed by Dr. Börner. This was followed by Mr. Bradford Paley, who discussed visual and cognitive engineering techniques that support the analysis of scientometric networks (


With the evident paradigm shift from print and paper to official and nonofficial online networks, as well as usage data and the wealth of data that they offer, the main discussion point during the symposium was the need for multidimensionality of measurements that capture and represent the complex arena we call “Scientific Impact”. Today, using one method, value or score to determine whether a researcher, research group or institution is indeed impactful seems invalid. If there is a lesson to be learned from this research event, it is that the scientific community has to find the correct and fair balance between a variety of computational metrics and qualitative peer-review processes.

Presentations & Audio recordings of this event are available:


  1. OECD (2010), Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings, OECD Publishing.
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)