Dr. Henk F. Moed
Centre for Science and Technology Studies (CWTS),
Leiden University, the Netherlands
moed@cwts.leidenuniv.nl

Bibliometric investigators - and other members of the scholarly community and research policy arena - are increasingly aware of the need to analyze and take into account the side-effects of bibliometric indicators when evaluating a scholars’ publication and referencing practices. Evidence of these effects is often informal, or even anecdotal, but recent studies have begun to examine these effects in a systematic way.

A longitudinal bibliometric analysis of UK science, covering almost 20 years, revealed three distinct patterns in scientists’ behavior. This was in response to the principal evaluation criteria applied in the Research Assessment Exercises (RAE) of 1992, 1996 and 2001 and was aimed at attaining the most favorable funding results (1). When total publication counts were requested for the 1992 RAE, UK scientists substantially increased their article production. Further evidence of this type of behavior was observed when a shift from 'quantity' to 'quality' in evaluation criteria was announced for the 1996 RAE; in response, UK authors gradually increased their number of papers in journals with a relatively high impact factor. Prior to the 2001 RAE, evaluated units shifted back from ‘quality’ to ‘quantity’, particularly by encouraging their members to collaborate or at least co-author more intensively, and thus increase the number of active research staff.

Sophisticated indicators based on citations are more informative of a group’s research performance and less easily manipulated than indicators based on the number of papers published in journals with a high citation impact factor. For instance, a high impact group can receive its citations from hundreds of different institutions. The distribution of citations among citing institutions is skewed, and the contribution of its tail is large. Making ‘citation trading’ arrangements with a few institutions will not have such a profound effect on citation counts as to significantly benefit an author’s reputation, and thus potentially also funding received.

Nevertheless, it cannot be claimed that such indicators are not affected by strategic behavior. I am very keen to be notified of cases of actual, or probable, strategic behavior by authors and journal editors directly aimed at influencing bibliometric indicators. When measuring methods are refined, researchers are likely to manipulate any shortcomings that arise. I would welcome any information on these shortcomings that may help improve those methods. Please feel free to contact me.

References:

(1) Moed, H.F. (2008) “UK Research Assessment Exercises: Informed judgments on research quality or quantity?” Scientometrics, Vol. 74, pp. 141-149.
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)