Issue 4 – March 2008

Articles

Assessing the Shanghai Rankings

The Academic Ranking of World Universities (ARWU), popularly known as the ‘Shanghai Rankings’, is one of the most cited university rankings in the world. We look at its evaluation criteria and how it is perceived by the academic community.

Read more >


The ranking of universities internationally has become more commonplace in recent years. This seems to be predominantly connected to competition and accountability. Students are increasingly moving across national borders and want to compare faculties and departments in different countries, universities want to attract the best teachers and researchers, and there is an increasing feeling that the public is entitled to know how institutes that benefit from public funds are performing (1).

The two most frequently cited university rankings are: the Academic Ranking of World Universities (ARWU), from Shanghai Jiao Tong University (SJTU) in China; and the World University Rankings, from the Times Higher Education Supplement (THES), a London-based weekly newspaper, in cooperation with its research and data analysis arm Quacquarelli Symonds (QS).

ARWU, often referred to simply as the ‘Shanghai Rankings’, was originally developed to compare Chinese universities with others worldwide, with particular reference to academic and research performance. The rankings, which cover 500 universities, including Tsinghua University in China, have been posted annually on the university’s website since 2003. THES has published its rankings annually since 2004. The assessment indicators and their weightings used in both rankings are outlined in the sidebar.

Academic debate

Both rankings have been the subject of considerable debate since their inception, garnering both positive and negative reactions from the academic community. “For the most part the [Shanghai Rankings] are methodologically sound and a valid basis for synchronic global comparisons,” said Professor Simon Marginson, Chair in Higher Education at the University of Melbourne, Australia in a paper delivered at a conference of the Asia-Pacific Association for International Education in Singapore in March 2007. Alex Usher, Vice President of the Educational Policy Institute, a US non-profit organization, commented on the institute’s website last November that he believes the Shanghai Rankings to be superior to the THES Rankings at the moment.

Science bias

However, others have criticized the Shanghai Rankings for being biased towards science-focused institutions because of the publication outlets considered and the extraordinary amount of citations in these fields. This is a bias that SJTU is aware of, as evidenced in a paper published by the Rankings’ founders in 2004: “Many well-known institutions specialized in humanities and social sciences are ranked relatively low partly because of the imbalances in the production of articles among various subject fields. The Ranking Group tried hard but was unsuccessful in finding additional indicators that are special for humanities and social sciences” (2).

Despite their flaws, however, and a concern that rankings promote a ‘one-size-fits-all’ approach to assessment, it is generally agreed among the academic community that these two rankings are the most comprehensive efforts available at present to rank universities internationally.

References:

(1) Holmes, R. (2006), “The THES Rankings: Are they really world class?”, Asian Journal of University Education, Vol. 1, No. 1, pp. 2.
(2) Liu, N.C., Cheng, Y. (2005) “Academic Ranking of World Universities – Methodologies and Problems”, Higher Education In Europe, Vol. 80, No. 2, pp. 10.

Shanghai Rankings (weighted scores)

Total number of staff (contributes 20% of the overall Ranking score) and alumni of institutions (10%) having won Nobel Prizes or Fields Medals

Number of highly cited researchers in 21 different disciplines (20%)

Number of articles published in Nature and Science (20%)

Total number of articles indexed by Science Citation Index and Social Science Citation Index (20%)

Research performance (total scores of the above) per head of staff (10%)

For definitions of indicators and further details, click here.

World University Rankings (THES)

Research quality (peer review 40%, citations per faculty 20%)

Graduate employability (recruiter review 10%)

International outlook (international faculty 5%, international students 5%)

Teaching quality (student faculty 20%)

For further details, click here.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
Research Trends Image

Australia: new government, new research opportunities

A new Australian government is already making an impact on research funding policies. Research Trends talks to Professor Alan Johnson AM about recent policy changes, as well as longer-term funding trends.

Read more >


With a new Labor government instated in Australia in December 2007, changes in research funding policy and procedure were expected. Now, four months into its term of office, the government has instigated a number of reviews of science and technology and is beginning to announce the impact that these will have.

When it comes to opportunities to attract research funding in Australia, there are at least 130 different funding schemes to which academics can apply via the Australian Competitive Grants Register (ACGR). However, universities also receive so-called ‘block funding’ from the government based on several factors (Figure 1), especially student profile; this includes student numbers and type of discipline.

The Australian Research Council (ARC) is an authority whose mission is to advance Australia's research excellence, to be globally competitive and deliver benefits to the community. In doing so, it advises the government on research matters and manages the National Competitive Grants Program (part of the ACGR), a significant component of Australia's investment in research and development. At the time of writing, the Australian government has suggested that it will preserve the independence of the ARC, for which it has already established an independent advisory council.

Meanwhile, the government has committed to creating a charter for public research agencies including the Commonwealth Scientific and Industrial Research Organisation, Australian Institute of Marine Science and the Australian Nuclear Science and Technology Organisation. The charter aims to identify the responsibilities of each organization to guarantee that they carry them out.

Promoting flexibility

But what do these changes mean for individual researchers and institutions? We ask Professor Alan Johnson AM, industry expert at Research Management Services International, to elucidate. He succinctly shares his own definitions of research and innovation: “I define research as turning money into knowledge, and innovation as turning knowledge into money.”

He goes on to explain how this applies to the situation in Australia: “The government aims to determine how the national innovation system (the flow of technology and information among people, enterprises and institutions which is key to the innovative process on the national level) should perform in order to improve both innovation and research. In practice, this means two major changes: firstly, there is likely to be a radical shift from centralized sectoral reform to mission-based compacts between the commonwealth and individual institutions. This will promote operational flexibility, and covers education, research and research training, community outreach and innovation. This will allow universities in particular to negotiate with the government to determine their own research priorities.”

Secondly, the Research Quality Framework (RQF) project has been abolished. The RQF was intended to be a national assessment of university research based around measuring quality and impact, similar to the UK’s Research Assessment Exercise. The government has suggested that it will use a more metrics-based system instead. While universities are able to put funds that they had allocated internally for administrating the RQF back into research, competition for funding from the ACGR schemes is likely to intensify.

Funding trends

While the overall amount of money allocated to research in 2008 is not expected to rise significantly from 2007 levels, there has been a shift in research priorities over the last seven years. Higher education research expenditure on commerce and management increased by 51% between 2001 and 2005, according to the Australian Bureau of Statistics. The expenditure on Earth Sciences, however, rose far less during the same period: 21% for Biological Sciences, 27% for Chemistry, 17% for Physics and 31% for Mathematical Science. “As a result,” Johnson concludes, “Australia may have a significant knowledge and expertise gap in science and technology in the coming years.”

Figure 1 - Apparent flows of funding and spending (in Australian dollars). Source: ‘Public support for science and innovation’ by Australian Government Productivity Commission, March 9, 2007.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Focus on China: the trajectory of Chinese research

Chinese article output has increased 18% per annum over the last 10 years, and the share of global articles with at least one Chinese author grew from 3% in 1997 to almost 13% in 2006. What is driving this growth and what effect is it having on the global research landscape?

Read more >


Since the invention of movable type printing by Bi Sheng almost 1,000 years ago, China has had a long tradition of disseminating the printed word. In the 21st century, the growth in scholarly journal articles by Chinese authors has been nothing short of prodigious. How can we track this growth, and what has its effect on the global research landscape been?

“China has pursued a program of modernization for over 30 years, in particular by increasing the contribution of scientific innovations to the economy,” says Sharon Ruwart, Managing Director of Science & Technology for Elsevier China. “This is helping the country move beyond agriculture and manufacturing into higher value-added production with more indigenous innovation.”

This policy focus has contributed to an exponential increase in Chinese article output of 18% per annum over the last 10 years (Figure 1). As a result, the share of global articles with at least one Chinese author has grown from 3% in 1997 to almost 13% in 2006. In 2006, 49% of these articles were published in English and 51% in Chinese, a ratio that has remained more or less stable over the last decade.

Figure 1 – Number of articles published by Chinese researchers in all languages (dark blue) and those in the Chinese language (light blue) 1997-2006. Source: Scopus

According to Ruwart, this rapid growth seems likely to continue: “The government clearly signalled the high priority it places on science by unveiling a 15-year plan (2006-2020) to systematically invest in designated fields of science and technology, with associated goals for each. One of the plan's key benchmarks is an increase in R&D expenditure from 1.4% to 2.5% of GDP. Underlying GDP growth is estimated to quadruple between 2000 and 2020.”

Other factors contributing to the dramatic increase in scholarly output in the last decade include government and university incentives to publish in international journals, increased exposure to the journal literature via online platforms since the late 1990s and expanded enrolment in higher research degrees since 2000.

The influence of Chinese research on the rest of the world can be gauged by looking at the most influential articles authored solely by authors based in China. The top 14 have collectively been cited more than 6,000 times to date (Table 1). However, according to Martin Tanke, Managing Director of Science & Technology Journals Publishing at Elsevier, “This table masks the quality gap we currently see between well-established international research and the typical low impact of many Chinese papers. But this is starting to change as China moves away from its focus on quantity alone.”

First author

Main affiliation

Article title

Year

Journal

Cites to Feb 2008

Yu J.

Beijing Genomics Institute, Center of Genomics and Bioinformatics, Chinese Academy of Sciences, Beijing

A draft sequence of the rice genome (Oryza sativa L. ssp. indica)

2002

Science

987

Han W.

Department of Physics, Center of Atomic and Molecular Sciences, Tsinghua University, Beijing

Synthesis of gallium nitride nanorods through a carbon nanotube- confined reaction

1997

Science

806

Shen Z.-X.

Shanghai Institute of Hematology, Ruijin Hospital, Shanghai Second Medical University, Shanghai

Use of arsenic trioxide (As2O3) in the treatment of acute promyelocytic leukemia (AFL): II. Clinical efficacy and pharmacokinetics in relapsed patients

1997

Blood

576

Guan Y.

Department of Microbiology, University of Hong Kong, Queen Mary Hospital, Hong Kong

Isolation and characterization of viruses related to the SARS coronavirus from animals in Southern China

2003

Science

451

Kong Y.C.

Department of Physics, Mesoscopic Physics National Laboratory, Peking University, Beijing

Ultraviolet-emitting ZnO nanowires synthesized by a physical vapor deposition approach

2001

Applied Physics Letters

391

Liu L.

State Key Laboratory of Engineering Plastics, Institute of Chemistry, Chinese Academy of Sciences, Beijing

Studies on Nylon 6/Clay Nanocomposites by Melt-Intercalation Process

1999

Journal of Applied Polymer Science

385

Fan E.

Institute of Mathematics, Fudan University, Shanghai

Extended tanh-function method and its applications to nonlinear equations

2000

Physics Letters SectionA

370

Chen G.-Q.

Shanghai Institute of Hematology, Ruijin Hospital, Shanghai Second Medical University, Shanghai

Use of arsenic trioxide (As2O3) in the treatment of acute promyelocytic leukemia (APL): I. As2O3 exerts dose-dependent dual effects on APL cells

1997

Blood

349

Lin B.

Structure Research Laboratory, Academia Sinca, University of Science and Technology, Hefei

Green luminescent center in undoped zinc oxide films deposited on silicon substrates

2001

Applied Physics Letters

331

Lo C.-M.

Center for the Study of Liver Disease, University of Hong Kong Medical Center, Queen Mary Hospital, Hong Kong

Randomized controlled trial of transarterial Lipiodol chemoembolization for unresectable hepatocellular carcinoma

2002

Hepatology

313

Luo H.

Department of Chemistry, Peking University, Beijing

Investigation of the electrochemical and electrocatalytic behavior of single-wall carbon nanotube film on a glassy carbon electrode

2001

Analytical Chemistry

311

Zheng S.-B.

Department of Physics, University of Science and Technology of China, Hefei

Efficient scheme for two-atom entanglement and quantum information processing in cavity QED

2000

Physical Review Letters

300

Feng L.

Center of Molecular Sciences, Institute of Chemistry, Chinese Academy of Sciences, Beijing

Super-hydrophobic surfaces: From natural to artificial

2002

Advanced Materials

289

Wang J.

College of Chemistry and Molecular Engineering, Peking University, Beijing

Direct electrochemistry of cytochrome c at a glassy carbon electrode modified with single-wall carbon nanotubes

2002

Analytical Chemistry

276

Table 1 – Top-cited articles published solely by Chinese researchers 1997-2006, with citations received to date. Source: Scopus

China’s traditional research strengths have been in Physics, Chemistry, Materials Science and Engineering, but recently its developing expertise in the Health and Life Sciences has begun to emerge (Figure 2). Tanke continues: “In China there is an enormous emphasis on applied science rather than pure science, as research is expected to deliver tangible benefits to society such as highways, dams, hybrid crops, satellite systems and vaccines.”

Figure 2 - Proportion of journal articles in selected subject areas published by Chinese researchers in 2006. Source: Scopus

Given the high hopes that science will help sustain the country's continued development, the coming years will continue to see China expand and deepen its research capabilities. This is not a temporary phenomenon; as a science power, China is here to stay.

To see the citation report of six countries, including China and Australia, please click here.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Social sciences literature in citation databases

Social scientists have traditionally published more often in monographs than journals, when compared to fundamental and applied science researchers. However, the last 40 years have seen a continuing trend towards publication in journals, resulting in more citation information for the social sciences being indexed in citation databases. Professor Charles Oppenheim assesses the databases with social sciences coverage.

Read more >


Scholarly communication in the social sciences differs from that in the pure sciences. Social scientists publish more often in monographs than journals, when compared to fundamental and applied science researchers. Monographs and their references are not systematically indexed in databases. It is estimated that journal articles account for 45-70% of research output in the social sciences, depending on the discipline (1). As a result, citation studies in these fields require additional care since they can give an incomplete and inaccurate representation of research output if they focus only on journal articles.

Professor Charles Oppenheim, an information scientist for 40 years, and currently Head of Information Science at Loughborough University, UK, has found, however, that there is a continuing trend in the social sciences to publish increasingly in journals. “After World War II, science was seen as successful, a paradigm: it cured diseases, created energy supplies and so on,” he says. “Social sciences felt a bit like Cinderella; they were left out of the funding and grammar of science. Subconsciously, social scientists thought that if they aped pure science, one way of which was to publish in journals, then they might be able to get a larger slice of the funding pie.”

Assessing social sciences output

He continues, “A much more conscious reason is things like the Research Assessment Exercise in the UK, the principal method by which university research funding decisions have been made since 1986. The RAE typically requires each individual who is being returned by a university for consideration to identify four of his/her publications for evaluation. If you’re working on a monograph between each assessment – which takes place roughly every four years - you won’t have four papers available, and producing four monographs in that time is unrealistic.” The RAE will take place in its present form for the last time this year, and it is expected that future assessments will be based, at least in part, on bibliometrics. This will make citation counts increasingly important for all the sciences.

So how does one analyze research output in the social sciences? In 2006, the Economic and Social Research Council (ESRC), a research funding and training agency in the UK, asked Professor Oppenheim to help it answer this question. “The ESRC was under pressure from the British government to come up with a measure of the quality of social sciences research conducted in the UK, compared to research done elsewhere. It could find no single database that supplied this information and so asked me to conduct research into the databases available and suggest which one would be the best to use for this study,” says Oppenheim. “Until quite recently, Thomson’s Web of Science (WoS) was the only credible database which had reasonable social sciences coverage and provided citation indexing. In the last years, CSA Illumina, Google Scholar and Scopus have also entered the market, offering a similar service. My research thus covered these four databases. Their holdings and citation records were assessed against two sets of data: one from the 2001 RAE, the other from the International Bibliography of the Social Sciences, a bibliography managed by the London School of Economics and Political Science.”

Analyzing the results

The results of the research have since been published in the Journal of Informetrics (2). They suggest that of the four databases studied, WoS and Scopus offer the best social sciences coverage at journal, article and cited reference level. Both have a comprehensive ‘cover-to-cover’ indexing policy, although Scopus’ coverage only captures references for documents published after 1995. In citation searches carried out for records published after 1995, Oppenheim found that there was a 5.4% advantage in Scopus’ favor. CSA Illumina fared best when it came to foreign language journal coverage.

“Despite Scopus’ limited coverage of foreign language journals, something I suggest it consider extending for goodwill purposes, my research concluded that Scopus, with good coverage and sufficient tools to analyze citation counts, is arguably the best choice of the four databases reviewed and could be used as an alternative to WoS to evaluate research impact in the social sciences.”

Professor Oppenheim’s full article and methodology can be found here.

References:

(1) Archambault, É., Vignola Gagné, É. (2004), “The use of bibliometrics in the social sciences and humanities”, Science-Metrix Report, pp.3
(2) Norris, M., Oppenheim, C. (2007) “Comparing alternatives to the Web of Science for coverage of the social sciences’ literature”, Journal of Informetrics, Vol. 1, No. 2, pp.161-169.
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Why did you cite…?

In this section, we ask authors what motivated them to cite certain references. This issue we talk to authors who cited Nobel Prize winners and ask whether winning the Nobel Prize has a positive effect on a scientist’s citation inflow.

Read more >


In 2001, the Nobel Prize in Physiology or Medicine was awarded to Leland H. Hartwell, R. Timothy Hunt and Sir Paul M. Nurse “for their discoveries of key regulators of the cell cycle”. The Sveriges Riksbank Prize in Economic Sciences in the same year was awarded to George A. Akerlof, A. Michael Spence and Joseph E. Stiglitz “for their analyses of markets with asymmetric information”.

The annual rate of increase in total citation inflow to the growing collection of these Nobel laureates’ papers did not change appreciably after 2001. But was there any change in the reason for these citations being made?

It does not appear so. Professor Kathleen Gould, of the Vanderbilt University School of Medicine, Nashville, USA comments that she cites Nurse’s work “because 1) some are also my papers and I cite my old work as background for my new work and 2) our work overlaps in subject matter. I haven’t changed my citation patterns since 2001”.

Professor Paul Russell of The Scripps Research Institute, La Jolla, USA echoes this: “There was no change for my reasons in citing his work after his receipt of the Nobel Prize. I was [already] well acquainted with his work, and he was already a highly respected and influential leader in my field of research, so the prize didn’t really change my citation pattern.”

The Nobel Prize also seems to have had no effect on the reasons for Stiglitz’ publications being cited, despite economics having very different citation characteristics from physiology and medicine. Professor Philip Arestis of Cambridge University states: “The reason I cited [him] is very obvious: he had undertaken and published relevant and good work; that is the reason and nothing else.”

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Did you know

New paper measures return on investment in libraries

Academic libraries are under growing pressure to demonstrate their value to their institutions. The question increasingly being asked is: for every dollar invested in the library, how many dollars does the university receive in return?

A case study conducted in 2007 at the University of Illinois at Urbana-Champaign (UIUC) by Judy Luther, President of Information Strategies at UIUC, set out to answer this question. The results were published in a Library Connect white paper in 2008. Rather than measuring time or resources saved, the approach taken by most cost/benefit studies, this study focused on grant income generated by faculty using library resources.

While significant work is being done in research evaluation, no existing models for calculating a return on investment in academic libraries were found. A model for the university environment was thus developed, inspired by an article by Roger Strouse of Outsell, Inc. (1). The results, which are very much a first step, showed that for every dollar invested in the UIUC library in 2006, $4.38 was generated in grant income for the university in return. The full paper can be found here.
(1) Strouse, R. (2003) “Demonstrating value and return on investment: the ongoing imperative,” Information Outlook, Issue 3, pp. 14-19.

6
https://www.researchtrends.com/wp-content/uploads/2011/01/Research_Trends_Issue4.pdf