What is research assessment, what purposes does it serve, and how is it carried out? Research Trends introduces the basic components of this complex process.
In upcoming issues of Research Trends we dedicate attention to research assessment. Here we explain why we have chosen this subject, how it is defined, its historical background, how the article series is built up, and which topics will be addressed. We also highlight a few fundamental principles that underlie the subsequent articles in the series.
Measuring returns on investment
Research assessment is a broad endeavour. At root it is an attempt to measure the return on investment in scientific-scholarly research. Research assessment includes the evaluation of research quality and measurements of research inputs, outputs and impacts, and embraces both qualitative and quantitative methodologies, including the application of bibliometric indicators and mapping, and peer review.
Research performance is increasingly regarded as a key factor in economic performance and societal welfare. As such, research assessment has become a major issue for a wide range of stakeholders, and there is consequently an increasing focus on research quality and excellence, transparency, accountability, comparability and competition.
Institutions compete for students, staff and funding through international rankings.
This focus means that government funding of scientific research – especially in universities – tends to be based more and more on performance criteria. Such a policy requires the organization of large-scale research assessment exercises by national governmental agencies. The articles in this issue are intended to provide a concise overview of the various approaches towards performance-based funding in a number of OECD member states.
The institutional view
Today, research institutions and universities operate in the context of a global market. International comparisons or rankings of institutions are published on a regular basis, with the aim of informing students and knowledge-seeking external groups about their quality. Research managers also use this information to benchmark their own institutions against their competitors.
In light of these developments, institutions are increasingly setting up internal research assessment processes, and building research management information systems. These are based on a variety of relevant input and output measures of the performance of individual research units within an institution, enabling managers to allocate funds within the institution according to the past performance of the research groups.
At the same time, trends in publishing have had a crucial impact on assessing research output. Major publishers now make all their content electronically available online, and researchers consistently report that their access to the literature has never been better. In addition, disciplinary or institutionally oriented publication repositories are being built, along with the implementation of institutional research management systems, which include metadata on an institution’s publication output. Currently, three large multidisciplinary citation indexes are available: Elsevier’s Scopus, Thomson Reuters’ Web of Science, and Google Scholar.
In conjunction with the increasing access to journals and literature databases, more indicators of research quality and impact are becoming available. Many bibliographical databases implement bibliometric features such as author h-indexes, as well as publication and citation charts. More specialized institutes produce other indicators, often based on raw data from the large, multi-disciplinary citation indexes. Today, the calculation of indicators is not merely the province of experts in bibliometrics, and the concept of “desktop bibliometrics” is increasing becoming a reality.
An overview of the various topics covered in this series is presented in Table 1 below. These topics will be presented in short review articles, illustrative case studies, and interviews with research assessment experts and research managers. The main principle underlying the various articles in this series is that the future of research assessment exercises lies in the intelligent combination of metrics and peer review. A necessary condition is a thorough awareness of the potentialities and limitations of each of these two broad methodologies. This article series aims to increase such awareness.
Table 1 — Overview of topics addressed