Issue 3 – January 2008

Articles

The RAE: measuring academic research

Higher education research assessment in the UK is changing. Carried out for over 20 years via a method known as the Research Assessment Exercise (RAE), 2008 will be the last time the RAE takes place in its present form. What is it and why is it changing?

Read more >


The Research Assessment Exercise (RAE) is the principal method used in the UK to establish the quality of the research undertaken in the higher education sector. Conducted jointly by the UK’s four principal funding bodies - the Higher Education Funding Council for England, the Scottish Funding Council, the Higher Education Funding Council for Wales and the Department for Employment and Learning, Northern Ireland - it is an exercise based on peer review and aims to produce quality profiles for each submission of research activity made by an institution. Any higher education institution eligible to receive research funding from one of these funding bodies is eligible to participate in the exercise. It has taken place in its present form in 1986, 1992, 1996 and 2001, in which year a more rigorous exercise was carried out.

Traditionally, the RAE’s main objective has been to allow funding bodies to assess the quality of research arising from the investment of public funds. It is used as a means for the academic sector to assess its success and prepare its future strategy. As such, the RAE introduces an incentive to individuals and institutions to improve their research performance and, unlike other forms of review and assessment in higher education, it has retained a good degree of support among academic staff.

The 2008 RAE will be based on the same principles of peer review as in previous years. For the purpose of this year’s assessment, each academic discipline is assigned to one of 67 units of assessment (UOA). Work submitted is then assessed against the published criteria by an expert panel drawn from higher education institutions and the wider research community. The results will be published as a graded profile for each UOA for each submission.

RAE versus REF

After 2008, the UK intends to change the way research quality is evaluated to a more statistics-based system. The main reasons put forth by the Higher Education Funding Council for England for this change are that the RAE is expensive and burdensome for the participating institutions. It is expected that bibliometrics will be central to judging research quality in this new system. The proposed method, known as the Research Excellence Framework (REF), will therefore rely more heavily on statistics than the extensive – but arguably costly - peer review of the current RAE.

The REF will consist of a framework for the assessment and funding of academic research that takes into account the key differences between disciplines. Research income, research student data and a new bibliometric indicator of research quality will drive assessment and funding decisions for the science-based disciplines. A form of peer review will remain in place for the arts, humanities and social sciences.

The REF is expected to be phased in gradually. It will inform funding for science-based disciplines from 2010, with all other disciplines following from 2014.

Research assessment elsewhere

The UK is not the only country with fresh plans for the allocation of research funding. Australia recently announced the final specifications of its first research evaluation method, the Research Quality Framework (RQF). While the RAE has a stronger emphasis on research environment and esteem indicators, the RQF grades both research quality and its impact. However, they also share common characteristics, namely similarities in the proportion of work to be examined, a minimum size of research grouping as well as rules for eligibility for inclusion in the volume count for funding. More details of the RQF can be found here.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
Research Trends Image

The effects of bibliometric indicators on research evaluation

Bibliometric investigators are becoming increasingly aware of the need to take the side-effects of bilbiometric indicators into account when evaluating a scholar’s publication and referencing practices. Dr. Henk Moed explains why.

Read more >


Dr. Henk F. Moed
Centre for Science and Technology Studies (CWTS),
Leiden University, the Netherlands
moed@cwts.leidenuniv.nl

Bibliometric investigators - and other members of the scholarly community and research policy arena - are increasingly aware of the need to analyze and take into account the side-effects of bibliometric indicators when evaluating a scholars’ publication and referencing practices. Evidence of these effects is often informal, or even anecdotal, but recent studies have begun to examine these effects in a systematic way.

A longitudinal bibliometric analysis of UK science, covering almost 20 years, revealed three distinct patterns in scientists’ behavior. This was in response to the principal evaluation criteria applied in the Research Assessment Exercises (RAE) of 1992, 1996 and 2001 and was aimed at attaining the most favorable funding results (1). When total publication counts were requested for the 1992 RAE, UK scientists substantially increased their article production. Further evidence of this type of behavior was observed when a shift from 'quantity' to 'quality' in evaluation criteria was announced for the 1996 RAE; in response, UK authors gradually increased their number of papers in journals with a relatively high impact factor. Prior to the 2001 RAE, evaluated units shifted back from ‘quality’ to ‘quantity’, particularly by encouraging their members to collaborate or at least co-author more intensively, and thus increase the number of active research staff.

Sophisticated indicators based on citations are more informative of a group’s research performance and less easily manipulated than indicators based on the number of papers published in journals with a high citation impact factor. For instance, a high impact group can receive its citations from hundreds of different institutions. The distribution of citations among citing institutions is skewed, and the contribution of its tail is large. Making ‘citation trading’ arrangements with a few institutions will not have such a profound effect on citation counts as to significantly benefit an author’s reputation, and thus potentially also funding received.

Nevertheless, it cannot be claimed that such indicators are not affected by strategic behavior. I am very keen to be notified of cases of actual, or probable, strategic behavior by authors and journal editors directly aimed at influencing bibliometric indicators. When measuring methods are refined, researchers are likely to manipulate any shortcomings that arise. I would welcome any information on these shortcomings that may help improve those methods. Please feel free to contact me.

References:

(1) Moed, H.F. (2008) “UK Research Assessment Exercises: Informed judgments on research quality or quantity?” Scientometrics, Vol. 74, pp. 141-149.
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

UK universities climb in THES rankings

The 2007 THES-QS World University Rankings, published in November, identified 32 UK universities among the top 200 world universities. We look at the UK institution whose ranking increased the most between 2006 and 2007 and discuss this increase with its top-cited author.

Read more >


In November 2007 the Times Higher Education Supplement (THES), in cooperation with QS (Quacquarelli Symonds), published its annual QS World University Rankings. Since its inception in 2004, this ranking system has developed into one of the most robust measures of comparative international university quality.

The 2007 rankings identify 32 UK universities among the top 200 from around the world. The top 10 of these UK universities are identified in Table 1 below.

Fig 1

Most of these top 10 universities show positive changes in their ranking between 2006 and 2007. Indeed, the University of Bristol increased 27 places, King’s College London increased 22 places and both University College London and the University of Warwick increased 16 places. However, these were not the highest climbers among the UK universities. Table 2 illustrates the UK universities that increased the most within the top 200.

THES made several key changes to its methodology last year. Perhaps the most significant of these is the use of Scopus data to compile the World University Rankings. This will certainly have had an impact on universities’ movement up and down the rankings, but there are many other contributing factors that could have affected the scores. With this in mind, we asked the top-cited author from Lancaster University, the university with the highest rank increase, why he thinks his institution performed so well.

The most cited article from Lancaster University between 2002-2006 was authored by Professor David Lyth and co-authored by Professor David Wands of the University of Plymouth (“Generating the curvature perturbation without an inflaton”, 2002, Physics Letters, Section B: Nuclear, Elementary Particle and High-Energy Physics, Vol. 524, No. 1-2, pp. 5-14). Professor Lyth, commenting on his university’s impressive jump in ranking, says that “there has been a change in the methodology of our research - more thorough peer review and no self-citing of one's own institution is permitted. In addition, we have seen strong growth in staff numbers; in the physics department we have seen academic and research staff numbers grow by nearly 50% in the past six years.”

This article has shown a strong presence for UK universities in the THES rankings. The UK has a continued trend for quality research output, which is often groundbreaking and opens up the possibility for many other studies in new areas. 2007 was the first year THES employed its new methodology and several institutes have shown promising jumps in its rankings, based in part on their research. Future rankings will show whether they can maintain and cement their leading international positions.

To find out more about why Professors Lyth and Wand's paper was cited, please also visit Why did you cite...?

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

What is the best way to assess academic research?

The Research Assessment Exercise (RAE) will be conducted in the United Kingdom for the last time this year. The results will determine funding for higher education institutions for the next five years. Bahram Bekhradnia, Director of think tank HEPI, discusses his views of the RAE and whether he thinks the proposed new Research Excellence Framework will result in better funding allocation.

Read more >


This year will be the last time the Research Assessment Exercise (RAE) is conducted in the United Kingdom. From 2009, this system for assessing research quality at higher education institutions will be replaced by the Research Excellence Framework (REF). Submissions for the 2008 RAE closed on November 30, but institutions will have to wait until December 2008 to find out the results that will determine funding for the next five years. In the meantime, we talk to Bahram Bekhradnia, Director of the independent think tank Higher Education Policy Institute (HEPI), about his views of the RAE and whether he thinks the REF will result in better funding allocation.

As in most areas where funding is required, the calls on research funding are far greater than the resources available. “As a result there has to be rationing. The highest quality research should receive the most funding,” Bekhradnia says. “One can’t assume that the most prestigious universities produce the best research, however, so there has to be a system in place to assess research quality. This is what the RAE sets out to do.”

Similarly, as in most areas where demand outstrips supply, the RAE has seen its share of what Bekhradnia calls “game playing” or exploiting the rules to one’s own advantage (see also Dr. Moed's analysis in Research Trends).This has manifested itself in hiring and research decisions designed to attain the most favorable funding results. The RAE is currently based on review of research by distinguished discipline-based panels. The REF will be much more statistics based. However, while there is currently quite some discussion on whether bibliometrics, and in particular citation analysis, is a suitable way to measure quality, there is little doubt that the RAE enables assessments of quality.

Time for change
If the RAE aims to assess research quality in order to allocate resources appropriately, and it seems to be achieving this aim, this begs the question of why the system is changing. The main reasons put forth by the Higher Education Funding Council for England are that the RAE is expensive and burdensome for the participating institutions. Bekhradnia is not yet entirely convinced the REF will be less so on either of these fronts but believes that if bibliometrics are to be used in future, they should not dominate the assessment process. “The UK has a strong research reputation, but we should be cautious of any system that will lead to a concentration of research strength at too small a number of institutions. The ideal assessment method would be peer review informed by bibliometrics.”

Downloadable versions of HEPI reports, which include analyses of the RAE and REF, can be found here.

For an overview of the RAE, click here.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Why did you cite…?

In this section of Research Trends, we talk to the authors of top-cited articles. This issue, we select papers from the two UK universities that had the highest ranking increase in the 2007 THES-QS World University Rankings: Lancaster and Surrey. We ask the authors why they think they were cited and why those who cited them chose to do so.

Read more >


The Times Higher Education Supplement has identified the Universities of Lancaster and Surrey as the most improved UK universities. These universities’ most cited articles between 2002 and 2006 are:

  • Lyth (Lancaster University) and Wands (University of Portsmouth), “Generating the curvature perturbation without an inflaton”, 2002, Physics Letters, Section B: Nuclear, Elementary Particle and High-Energy Physics, Vol. 524, No. 1-2, pp. 5-14.
  • Ioannides (University of Surrey), “Pharmacokinetic interactions between herbal remedies and medicinal drugs”, 2002, Xenobiotica, Vol. 32, No. 6, pp. 451-478.

Opening up new research areas is a strong theme for UK research and something which is echoed by Professor Costas Ioannides from the University of Surrey, the institute showing the second highest increase in ranking. Professor Ioannides indicated that his paper “was the first published review to address herb-drug interactions from a mechanistic, rather than a descriptive, point of view. In fact, this is what prompted me to write this review in the first place.”

Professor David Lyth, commenting on his own paper, said, “I believe it was highly cited because it opened up a completely new possibility as to why structure exists in the Universe. We pointed out a completely different type of quantum fluctuation, which could lie dormant until a much later era. We called this fluctuation a ‘curvaton'. It has opened up many new possible avenues of research, and both the name and the idea have been taken up by many people.”

One of the scientists who has cited Professor Lyth's paper several times is Dr. Marieke Postma of the FOM-Institute of Subatomic Physics NIKHEF, Amsterdam, The Netherlands. When asked why she had cited this article, she said, “They came up with a new and original idea. Up until that point the usual lore was that the field responsible for inflaton was the same as that generating the observed density perturbations. This paper said, no, not necessarily, and gave an explicit scenario in which some other field (the "curvaton" field) was creating the density perturbations instead. The curvaton scenario opened up new ways of thinking. [We became] intrigued by that, and started exploring the consequences…”

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Did you know

How to reserve your place in history

In 1900, Moses Gomberg established himself as the grandfather of organic radical chemistry by describing the highly reactive triphenylmethyl (trityl) radical. Gomberg ended his landmark paper on this topic by saying, “This work will be continued and I wish to reserve the field for myself” (1).

Happily, this circumscription was readily ignored by subsequent generations, and organic radicals are now used in the production of a wide variety of plastics and synthetic rubber. Advances in organic radical chemistry throughout the 20th century have been key to our understanding of oxidative processes in normal living cells and in diseases such as cancer. Gomberg died in 1947, but his 1900 paper on the trityl radical remains a cornerstone of the field and was cited almost 100 times in the last decade, and 14 times in 2007 alone (data from Scopus).

(1) Gomberg, M. (1900) “An instance of trivalent carbon: Triphenylmethyl”, Journal of the American Chemical Society, Vol. 22, No. 11, pp. 757-771.

5
https://www.researchtrends.com/wp-content/uploads/2011/01/Research_Trends_Issue3.pdf
  • Elsevier has recently launched the International Center for the Study of Research - ICSR - to help create a more transparent approach to research assessment. Its mission is to encourage the examination of research using an array of metrics and a variety of qualitative and quantitive methods.