The Centre for Science and Technology Studies at Leiden University has developed a new research program focusing on monitoring and analyzing knowledge flows and on research evaluation. The program, which will be published this Fall, introduces new approaches to these well-established goals of scientometric research. With the development of this new program, first, we move from data-centric methods justified by ad-hoc reasoning towards a systematic theory-based framework for developing bibliometric and scientometric indicators. Second, in interpreting and applying performance indicators we increasingly base ourselves on the systematic analysis of current scientific and scholarly practices rather than only on general statistical arguments. Specific attention is paid to humanities and social sciences because of the variety of its research and publication practices. We also analyze the impact of research assessment exercises, and the performance criteria applied, on the primary process of knowledge production. Third, we explore the possibilities and problems in assessing the societal impact of research (“social quality”). Increasingly, this dimension is becoming the second pillar of research evaluation next to scientific impact and is creating a new challenge for science evaluation and assessment.

To sum up, we maintain the tried and trusted CWTS focus on bibliometrics for research evaluation, but we deepen our theoretical work and increase our empirical scope. Our new research agenda is a response to the widespread use of bibliometrics in performance based research management. We hope it will help prevent abuse of performance measures and thereby contribute to the development of good evaluation practices. We aim to bring scientometrics to a new level of quality in close collaboration with our colleagues in the field. This should also lead to new international standards of quality for assessments and science & technology indicators.

Figure 1 - Paul Wouters at a workshop of the Russian Academy of Sciences in St Petersburg, titled “Career Development in Academia”, 5–6 June 2012.

Research question

How can we improve our understanding of the dynamics of science, technology, and innovation by the measurement and assessment of the scientific and scholarly system, in particular of scientific products, communication processes and scholarly performance? This is the overarching theme of the new research program. In response, two specific research questions are in focus:

  1. How do scientific and scholarly practices interact with the “social technology” of research evaluation and monitoring knowledge systems?
  2. What are the characteristics, possibilities and limitations of advanced metrics and indicators of science, technology and innovation?

Key research themes

The first research theme in the program is the methodology of bibliometrics. Both at CWTS and elsewhere, the development of bibliometric indicators for research assessment has long been done in a pragmatic way. Indicators were developed without explicitly incorporating them in a broader mathematical or statistical framework. Indicators were justified mainly using empirical arguments. This resulted in a data-centric approach where the interpretation of the chosen indicators was developed in an ad-hoc fashion. In the new program we move towards a theory-oriented approach; indicator development will become more and more based on explicit theoretical models of the scientific publication and citation process. In this framework, the indicators will be judged on their mathematical and statistical properties. These models will for instance allow us to distinguish between observable and non-observable features of the publication and citation process (e.g., between the observable concept of citation impact and non-observable concepts such as scientific influence or quality). Model-based indicator development has the advantage of making an explicit distinction between what one intends to measure and what one is in fact measuring. This helps us to study the properties of bibliometric indicators (e.g., validity and reliability or bias and variance) in a more formalized way. The limitations of the indicators should be made explicit as well. For example, a complex concept such as scientific impact cannot be measured by one indicator. This is the reason we have moved from emphasizing one indicator (e.g. “the crown indicator”) towards a portfolio approach to performance indicators.

The new program also pays increasing attention to bibliometric network analysis and science maps. Bibliometric networks are networks of, for instance, publications, journals, researchers, or keywords. Instead of focusing on the properties of individual entities in a network, bibliometric network analysis concentrates on the way in which relations between entities give rise to larger structures, such as clusters of related publications or keywords. In this sense, bibliometric network analysis is closely related to the analysis of complex systems. The main objective of our research into bibliometric network analysis will be to provide content and context for research assessment purposes. Science maps enable us to analyze both the citation impact of a research group and its relationships with other groups. It also enables the analysis of interdisciplinary research without having to rely on predefined subject classifications. An interesting application is the visualization of the actual field profiles of research groups and scientific journals. We can also map the citation networks of journals at all levels of aggregation (see Figure 2).

Figure 2 - A map of journals based on citation relations. More maps can be found at

The second research theme in the program relates to the way evaluation processes configure the primary process of knowledge creation. The key question is that of the relationship between peer review based and indicator based evaluation. In the past, CWTS has dealt with this tension in a pragmatic way, using indicators to provide useful information to supplement peer review. As explained earlier, we will move towards a more systematic, theory based, approach in which we will probe in much more detail how expertise develops in particular scientific fields in relation to the bibliometric insights of those fields. We will not assume that the two ways of evaluating the quality of scientific and scholarly work are diametrically opposed: this would amount to setting up a straw man. In practice, peer review and bibliometrics are combined in a variety of ways. But how these combinations are developed by both evaluating institutions and the researchers that are being evaluated is not self-evident. Because it is exactly this interplay where the criteria for scientific quality and impact are being developed, we zoom in on this aspect of research evaluation.

Research evaluation may take different forms: annual appraisal interviews, institutional research assessment exercises, and global assessments of national science systems. Evaluation is a more complex interaction than simply the measurement of the performance of the researcher. We see it as a communication process in which both evaluators and the researcher under evaluation define what the proper evaluation criteria and materials should be. Therefore, we are especially interested in the intermediate effects of the process of evaluation on the researcher, evaluator, and on the development of assessment protocols.

Within this theme specific attention is paid to the “constructive” effects of research evaluation (including perverse effects). Evaluation systems inevitably produce (construct) quality and relevance as much as they measure it. This holds both for indicator based evaluation and for qualitative peer review evaluation systems. Evaluation systems have these effects because they shape the career paths of researchers and because they form the quality and relevance criteria that researchers entertain. These feedback processes also produce strategic behavior amongst researchers which potentially undermines the validity of the evaluation criteria. We therefore place focus on how current and new forms of peer review and indicator systems as main elements of the evaluation process will define different quality and relevance criteria in research assessment, on the short term as well as on the longer term. The recent anxiety about perverse effects of indicators such as the Hirsch-index will also be an important topic in this research theme. This theme will also encompass a research program about the development of scientific and scholarly careers and academic leadership.

Questions regarding the socio-economic and cultural relevance of scientific research form our third research theme. From the perspective of the knowledge-based society, policy makers stress the importance of “knowledge valorisation”. This term is used for the transfer of knowledge from one party to another with the aim of creating (economic and societal) benefits. However, the use of the word is often limited: only describing the transfer of knowledge to the commercial sector. The value in other domains, for example in professional or public domains, is often not taken into account. Also, the term valorisation is often used to describe a one-way-interaction, the dissemination of scientific knowledge to society, while in practice we often observe more mutual, interactive processes.

Within this research theme, we will therefore use the concept of “societal quality” in analyzing the societal impact of research. “Societal quality” is described as the value that is created by connecting research to societal practice and it is based on the notion that knowledge exchange between research and its related professional, public and economic domain strengthens the research involved. This definition encompasses explicitly more than economic value creation only. It also entails research that connects to societal issues and interactions with users in not-for profit sectors such as health and education as well as to the lay public. In the program we focus on the development of robust data sets, as well as the analysis of these datasets, in the context of specific pioneering projects in which the interaction between research and society can be well defined. This will create the possibility to construct, measure, and test potential indicators of societal impact.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)