On June 15, 2011, the European University Association (EUA) made public the results of the report ‘Global University rankings and their impact’. This report, led by Professor Andrejs Rauhvargers, provides a comparative analysis of the methodologies used in the most popular rankings*. The presentation of the report’s results was followed by a panel discussion with university leaders and higher education experts** about the impact of ranking on universities. The report does not intend to rank their various rankings but to analyze the methodologies and indicate the current developments of alternatives to measure university quality and performance in all its dimensions and complexity.
The authors of the report recognize that rankings are here to stay, given their high level of acceptance by various stakeholders. The report acknowledges the positive aspects of the rankings for universities: they draw the attention of governments to higher education and research; they improve accountability and management methods; and they demonstrate the importance of collecting reliable data. Regarding the robustness of the data on the output, both Web of Science and Scopus were mentioned as reliable databases as far as the sciences and medicine are concerned.
Main findings and criticisms
Going through the comparison of the various methodologies, the report details what is actually measured, how the scores for indicators are measured, and how the final scores are calculated — and therefore what the results actually mean.
The first criticism of university rankings is that they tend to principally measure research activities and not teaching. Moreover, the ‘unintended consequences’ of the rankings are clear, with more and more institutions tending to modify their strategy in order to improve their position in the rankings instead of focusing on their main missions.
For some ranking systems, lack of transparency is a major concern, and the QS World University Ranking in particular was criticized for not being sufficiently transparent.
The report also reveals the subjectivity in the proxies chosen and in the weight attached to each, which leads to composite scores that reflect the ranking provider’s concept of quality (for example, it may be decided that a given indicator may count for 25% or 50% of overall assessment score, yet this choice reflects a subjective assessment of what is important for a high-quality institute). In addition, indicator scores are not absolute but relative measures, which can complicate comparisons of indicator scores. For example, if the indicator is number of students per faculty, what does a score of, say, 23 mean? That there are 23 students per faculty member? Or does it mean that this institute has 23% of the students per faculty compared with institutes with the highest number of students/faculty? Moreover, considering simple counts or relative values is not neutral. As an example, the Academic Ranking of World Universities ranking does not take into consideration the size of the institutions.
Other indicators measuring teaching quality are perceived as strongly questionable, far more so than the ones measuring research. Moreover, the EUA report describes how differences in the way academics publish and cite each other in different fields can create a strong bias in rankings. As such, attempts have been made to normalize across disciplines, and the field-normalization in the Leiden Ranking is a highly regarded example of this effort.
The EUA report makes several recommendations for ranking-makers, including the need to mention what the ranking is for, and for whom it is intended. Among the suggestions to improve the rankings, the following received the greatest attention from the audience:
- Include non-journal publications properly, including books, which are especially important for social sciences and the arts and humanities;
- Address language issues (is an abstract available in English, as local language versions are often less visible?);
- Include more universities: currently the rankings assess only 1–3% of the 17,000 existing universities worldwide;
- Take into consideration the teaching mission with relevant indicators.
Which ranking, which evaluation tool for which purpose?
Going further, the panel discussion’s participants recommend going beyond the rankings and analysing in detail what information is needed by institutions to assess the diversity of research activities, and to take strategic decisions and implement those choices. Citation analysis or any other single indicator is obviously not sufficient to make decisions on a well-informed basis. The discussion should be to determine what the best tools are depending on the question.
Expectations about the U-Multirank project are high, considering its aim to show the various missions of the universities, far away from a league table, helping students to make choices.
As a conclusion, Jean-Marc Rapp, President of EUA, outlined the next steps in the EUA’s agenda: to analyze both the desired and un-intended consequences of the ranking systems, and comparing the different ways of assessing universities (ranking/rating, benchmarking, quality assessment and so on). This proves again how crucial those evaluation matters are for universities, who are looking for advice about how to make the most informed choices.
The original report: http://www.eua.be/pubs/Global_University_Rankings_and_Their_Impact.pdf
*The EUA looked at the following rankings: Shanghai Academic Ranking of World Universities (ARWU); Times Higher Education World University Ranking (in cooperation with Quacquarelli Symonds until 2009, and Thomson Reuters from 2010); World’s Best Universities Ranking (QS); Global Universities ranking (Reitor); HEEACT Rankings (Taiwan); EU University-based Research Assessment (AUBR Working Group, EU); Leiden Ranking; CHE University/Excellence rankings; U-Multirank; U-Map classification; and Webometrics.
**Participants involved in the meeting: 1) Presentation of the report: Jan Truszczynski, Director General for Education & Culture, European Commission; Allan Päll, Vice-Chairperson, European Student’s Union; Gero Federkeil, Vice President (VP) International Observatory on Academic Rankings and Excellence. 2) Panel discussion: Chaired by Professor Ellen Hazelkorn, VP Research & Enterprise, Dublin, Ireland; Professor Jean-Pierre Finance, President Université Henri Poincaré; Sir Howard Newby, Vice-Chancellor, University of Liverpool, United Kingdom; Jens Oddershede, Rector, University of Southern Denmark, Chairman of Universities Denmark.
Curriculum Vitae: Andrejs Rauvargers
Andrejs Rauhvargers was born 1952 in Riga, Latvia, and has a Ph.D. in Chemistry from the University of Latvia. He is Secretary General of the Latvian Rectors’ Conference and Professor at the Faculty of Education at the University of Latvia. He has also served as Deputy State Secretary at Latvian Ministry of Education, where he participated in developing legislation for higher education and was closely involved in the establishment of the higher education quality assurance system in Latvia and its coordination with neighboring countries Estonia and Lithuania. He was also responsible for establishing a system for academic recognition in Latvia.
Internationally, Rauhvargers is a member of the Bologna Follow-Up Group and since 2005 has chaired the working group studying the progress in the 46 ‘Bologna’ countries and preparing the Bologna Stocktaking reports published in 2007 and 2009. He served as president of the European Network of Academic Recognition Centres (ENIC) from 1997 to 2001 and as President of the Intergovernmental Committee of the Lisbon Recognition Convention from 2001 to 2008.
He is the author of several other major reports and a number of publications on various aspects of international higher education, both national and international, and has participated in and managed several higher education reform projects in Croatia, Latvia, Lithuania, Montenegro and Poland. He has been an invited speaker in his field in more than 20 countries. In 2006 Andrejs Rauhvargers was awarded the European Association for International Education’s Constance Meldrum Award for innovation, leadership, and inspiration in international higher education.