Professor Börner is also a curator of the Places & Spaces: Mapping Science exhibit currently on display at Northeastern University in Boston, MA (see http://scimaps.org). The exhibit is a collaborative work between Börner and researchers in diverse disciplines including scientometrics, network science, geography, education, and information visualization. Together, they design maps of science which introduce unique visualization tools that capture knowledge and enable deeper understanding of global scientific, environmental and economic trends to name a few.

We spoke with Katy Börner in order to better understand the manner by which she develops the different iterations of the exhibit and to gain some insight into the future of visualization in the era of ‘Big Data’ analysis.

How do you establish the collaborations with the different researchers featured in the different maps?

Each year, a call for maps is issued. This year’s Call for Maps for the 8th Iteration of the Places & Spaces: Mapping Science Exhibit on ‘Science Maps for Kids’ (2012) is at http://scimaps.org/call (Note: This call has now closed.) Map makers from many different countries and different areas of science submit individually or in teams. Submissions are carefully reviewed by the advisory board and external reviewers with expertise on the topic of the iteration. In 2012, we will invite children to serve as reviewers — if they cannot understand and make use of a map then this map will not be on display in the exhibit.

How important is the international aspect of the maps?

Today’s science is global and it has to be studied, mapped, understood, and managed globally. We welcome maps in all languages and from all countries and cultures.

Could you talk us through the various iterations of these maps?

The exhibit is a 10-year effort. Each year, 10 new maps are added resulting in 100 maps total in 2014. Iteration themes are as follows:

• 1^st Iteration (2005): The Power of Maps
• 2^nd Iteration (2006): The Power of Reference Systems
• 3^rd Iteration (2007): The Power of Forecasts
• 4^th Iteration (2008): Science Maps for Economic Decision Makers
• 5^th Iteration (2009): Science Maps for Science Policy Makers
• 6^th Iteration (2010): Science Maps for Scholars
• 7^th Iteration (2011): Science Maps as Visual Interfaces to Digital Libraries
• 8^th Iteration (2012): Science Maps for Kids
• 9^th Iteration (2013): Science Maps for Daily Science Forecasts
• 10^th Iteration (2014): Science Mapping Standards

 What are some of the guidelines that were developed for the iterations?

Each iteration compares and contrasts four existing maps (e.g., early cartographic maps in the first iteration on ‘The Power of Maps’) to six maps of science. Each map has to fit the theme of the respective iteration. Submissions are evaluated in terms of two kinds of values: i) Scientific value, which centers on quality of data collection, analysis and communication of results in support of clearly stated objectives, and whether the map represents appropriate and innovative application of existing algorithms and/or development of new approaches; ii) Value for user groups (e.g., kids), which considers the following questions: what major insight does the map provide and why does it matter? Is the map easy to understand? Does it inspire kids to learn more about science and technology?

Could you say a few words about the new iteration?

The 8^th iteration of the Mapping Science exhibit is devoted to science maps that kids aged 5–14 can use to gain a more holistic understanding and appreciation of science and technology. Each map should be engaging and fun to peruse yet should have at least one concrete learning objective. Among others, the maps might depict:  

• A concept map telling a science story;
• Famous adventures, encounters, or discoveries in science history;
• Zooms in-out of the world of science;
• Surprising, scary, wonderful, and exciting scientific activities;
• Timelines of science and technology development and inventions;
• Exhibit holdings at different science museums (location, subject matter, or both);
• A map of school science curricula, projects, or science textbook contents;
• Career trajectories in science;
• Science maps drawn by kids analogous to Children Map the World

 Maps are intended to give children the exciting opportunity to immerse in, explore, or navigate the landscape of science and to find their own place.

Places & Spaces: Mapping Science is a travelling exhibition. Can any interested party purchase their own maps and display them?

Maps are on display at public libraries, science museums, national academies of science, universities, companies and are seen, discussed, admired, and purchased by scholars, practitioners, educators, and others. Maps are available for sale at http://scimaps.org/store and proceeds help finance the creation of new iterations and their display at public venues.

How do you think ‘Big Data’ computation will influence the production and usefulness of scientific maps?

The bigger and more complex the datasets, the higher the need for effective data mining and visualization to guide data management, navigation, and utilization. Originally, publication, funding, and patent data were dominantly used in science of science studies. Today, researchers also study science news, job market, and even S&T twitter data—in real time. The monitoring, mining, modeling, and visualization of all relevant data streams in many different languages and the wide-spread distribution of results will require major computational infrastructures.

Is the visualization technology there to deal with big-data issues such as size and complexity?

The total number of scholars, papers, books, patents, and grants in existence today is rather small when compared with datasets generated and mined in medicine, physics, or meteorology. Tools like the Network Workbench (NWB) or Science of Science (Sci2) tool can extract, analyze, and layout networks with a million nodes. The NIH Map Viewer lets you interactively browse multiple years of funding by the National Institutes of Health. The mapping sustainability project maps seven different paper, patent, and funding datasets in geospatial and topic space (http://mapsustain.cns.iu.edu). 

Do you feel that the explosion in software solutions for network data management and visualization (including Network Workbench) has made life easier or harder for network scientists and in what way?

The Network Workbench was designed for researchers, educators, and practitioners interested in the study of biomedical, social and behavioral science, physics, and other networks and has been downloaded by 110,000 users around the globe. The Science of Science (Sci2) Tool was developed for science policy makers and researchers that study science by scientific means. It supports temporal, geospatial, topical, and network analysis and visualization of scholarly datasets at the micro (individual), meso (local), and macro (global) levels, and is used by many scholars, practitioners, and major agencies such as the National Science Foundation, the National Institutes of Health, the US Department of Agriculture, and the National Oceanic and Atmospheric Administration. While the agencies cannot share their data holdings, they can apply the very same analysis workflow to their internal data and compare results across agency boundaries. Plus, program officers and analysts can use Sci2 to download and analyze relevant data from, for example, Elsevier’s Scopus or Thomson Reuter’s Web of Science databases. Instead of getting a report from a contractor they now have an easy-to-use tool to answer their very own questions.

For more information about the exhibit please contact:

Prof. Katy Börner

School of Library & Information Science
Indiana University, Bloomington
Email: katy@indiana.edu

In order to advance the understanding and support education on the power of scientific mapping and visualization, Elsevier has purchased two sets of maps from the Mapping Science Exhibit.

 Each set contains nine maps that were selected from different iterations. The sets contain three scientific trends maps, three technology related maps, and three social/environmental maps.

 We at Elsevier would like to offer free shipment of the maps to Elsevier’s Research4Life eligible universities and research institutions around the world that are interested in displaying them on a temporary basis. Training and educational seminars will be available to interested parties.

To check your institution eligibility please visit: http://www.research4life.org/about.html  

For further information about the traveling exhibit please contact:

Gali Halevi. MLS., PhD
Director of Government Segment Marketing
Email: g.halevi@elsevier.com
Phone: 646-248-9464

In the years immediately following the end of hostilities in the Second World War, large numbers of highly skilled scientists emigrated from Western Europe to the United States. In the UK, concerns over the ‘loss’ of British researchers began to be raised in the early 1950s, as the weight of anecdotal (and limited direct) evidence began to mount. By the early 1960s the issue had become politicized and the Royal Society was tasked with reporting on the nature and extent of the problem. Their report, ‘Emigration of scientists from the United Kingdom’, was published in 1963 and received much media attention, but it was the Evening Standard newspaper that subsequently coined the term that was to encapsulate the concept: ‘brain drain’¹.

 Over time, the concept of brain drain has shifted in meaning and complexity, and is now generally understood to describe the shift of researchers from any country (typically less scientifically developed) to any other (typically more scientifically developed). Brain drain, as fits the negative connotations of the term, was usually considered as a win-lose scenario.

New models, new approaches

In recent years, the theoretical framework surrounding scientific mobility and migration has become sufficiently developed to require the coinage of a new term: brain circulation². According to this concept, nations are not considered as winners or losers but as loci in a dynamic system of human capital flows. Within this system, countries may accrue benefits to their domestic scientific capacity through diaspora effects (where the knowledge, skills and professional networks established by emigrant researchers while abroad are shared with colleagues at home) and return rates (where emigrant researchers return to their home countries after a period of working abroad, bringing with them the experiences they have gained)³. Such benefits are intangible and as such are difficult to quantify.

 Methodologically, studies of brain circulation have traditionally drawn on census or migration data², surveys of researchers^4,5, CV analysis^6,7, or a combination of methods⁸. However, empirical data showed that brain circulation cannot be modeled as a purely random process, since there are barriers of language, politics, culture and so on that may act to encourage or prevent a given researcher from moving to a given country. Another more recent study offered the interesting approach of using job advertisements posted on the website of a well-known science weekly to measure brain circulation, but showed that selection bias in the advert placements ruled out the broad applicability of this method⁹.

 With the advent of comprehensive and sophisticated online publication databases that are populated with peer-reviewed articles with complete author affiliation (address) data, new possibilities have opened up for wide-ranging studies of brain circulation. The development of a methodological framework using these databases was pioneered by Dr. Grit Laudel, currently at the University of Twente in the Netherlands. In her 2003 article ‘Studying the brain drain: can bibliometric methods help?’¹⁰, she presented the first systematic attempt to use authors’ listed addresses in published articles as a proxy for their location, so allowing tracking of their migration patterns over time. This study presented preliminary results demonstrating a net movement of ‘elite’ researchers to the US from the rest of the world (in a single specialty, angiotensin research).

 Using the same approach, Laudel subsequently expanded her study to demonstrate that while elite migration to the US can be found at the level of individual specialties (such as angiotensin research), the proportion of elite researchers in the US remained almost constant in the period 1980–2002¹¹. This finding across all subject fields appears to mask great lower-level variability, as Laudel demonstrates by contrasting the net gain over time of elite researchers by the US in angiotensin research with the relatively steady-state, US-centric elite researcher population of the vibrational spectroscopy community. Migration rates are therefore also likely to vary considerably at lower levels of aggregation than an entire country, such as at region, state, city or institution level.

Designing a novel approach to brain circulation mapping

As part of the report ‘International Comparative Performance of the UK Research Base: 2011’, commissioned by the Department for Business, Innovation and Skills (BIS), a fresh way of looking at researcher mobility was sought. In the report, published in October 2011, the Scopus database was used to produce a conceptual map of the stocks and flows of human capital in the UK over the 15-year period 1996–2010 (results detailed in Part II of this article in the next issue).

 In an important departure from previous studies using author affiliation data as a proxy for measuring brain circulation, this work was not confined to authors belonging to an elite or to a single subject or specialty (c.f. Refs 12–13). Instead, the approach presented in the report uses Scopus author profile data to derive a history of an author’s affiliations recorded in their publications and to assign them to mobility classes defined by the type and duration of observed moves. There were several conceptual and methodological issues to be resolved before the map could be built:

1. How can we unambiguously assign articles to their authors?

A longstanding problem in researcher mobility studies has been the unambiguous identification of the individual14, as there are common family names in every language and country, and multiple variants of a given person’s name in the published literature. In order to overcome these problems, Scopus has improved its author-profiling algorithm in order to identify individual researchers precisely. The Scopus Author Identifier gives each author a separate ID and groups together all the documents written by that author, matching alternate spellings and variations of the author’s last name and distinguishing between authors using sophisticated algorithm based on data elements associated with the article (such as affiliation, subject area, co-authors and so on).

2. What is a ‘UK researcher’?

Author nationality is not captured in article or author profiling data, and there are serious methodological difficulties in using cultural indicators (such as family names) as a proxy for nationality of birth¹⁵. So for this study, authors were assumed to be from the first country from which they have published, or from the country where they published the majority of their articles, when looking at migratory or transitory mobility respectively (see point 4 below). These criteria may, in individual cases, result in authors being assigned to migratory patterns that may not accurately reflect the real situation, but such errors may be assumed to be evenly distributed across the groups and so the overall pattern remains valid. To define the initial population for study, UK authors were identified as those that had listed a UK affiliation on at least one publication (articles, reviews and conference papers) published across the 18,000 journals included in Scopus during the period 1996–2010. This list included about 1.5 million unique authors.

3. What is an ‘active researcher’?

The 1.5 million UK researchers identified includes a large proportion of authors with relatively few publications (with UK or non-UK affiliations) over the entire 15-year period of analysis. As such, it was assumed that they are not likely to represent career researchers, but individuals who have left the research system. As such, a productivity filter was put in place to restrict to those authors with at least 1 article in the latest 5-year period (2006–2010) and at least 10 articles in the entire 15-year period (1996–2010), or those with fewer than 10 articles in 1996–2010 but more than at least 4 articles in 2006–2010. After applying the productivity filter, a set of 210,923 active UK researchers was defined and formed the basis of the study.

4. How should long- and short-term mobility be defined?

The study of brain circulation is complicated by the difficulties in teasing apart the related phenomena of long-term migration from short-term mobility (such as doctoral research visits, sabbaticals, secondments and so on), which might be deemed a form of collaboration. Defining a time period for a stay abroad over and above which it should be considered a permanent migration (migratory mobility), and below which should be deemed a short-term research visit (transitory mobility), is difficult. Drawing on the definition by Crawford et al.¹⁶, stays abroad of 2 years or more were considered migratory and were further subdivided into those where the researcher remained abroad or where they subsequently returned to their original country. Stays abroad of less than 2 years were deemed transitory, and were also further subdivided into those who mostly published under a UK or a non-UK affiliation. Researchers without any apparent mobility based on their published affiliations were treated as a separate group.

5. What indicators were applied to understand the groups better?

To better understand the composition of each group defined on the map, two aggregate indicators were calculated for each to represent, in a relative sense, the publication productivity and seniority of the researchers they contain. Relative Productivity represents a measure of the articles per year since the first appearance of each researcher as an author during the period 1996–2010, relative to all UK researchers in the same period, while Relative Seniority represents years since the first appearance of each researcher as an author during the period 1996–2010, relative to all UK researchers in the same period. Both Relative Productivity and Relative Seniority are calculated for each author’s entire output in the period (i.e., not just those articles listing a UK address).

References

1. Balmer, B., Godwin, M. & Gregory, J. (2009). The Royal Society and the ‘brain drain’: natural scientists meet social science. Notes Rec. R. Soc. 63, pp. 339–353.
2. Johnson, J.M. & Regets, M.C. (1998). International mobility of scientists and engineers to the United States—brain drain or brain circulation? Issue Brief (National Science Foundation), No. [?], pp. 98–316.
3. Ciumasu, I.M. (2010). Turning brain drain into brain networking. Science and Public Policy, Vol. 37, pp. 135–146.
4. Marceau, J. et al. (2008). Innovation agents: the inter-country mobility of scientists and the growth of knowledge hubs in Asia. Paper presented to the 25^th DRUID conference, Copenhagen, June.
5. Auriol, L. (2010). Careers of doctorate holders: employment and mobility patterns. OECD Science, Technology and Industry Working Papers doi: 10.1787/5kmh8phxvvf5-en
6. Dietz, J.S. et al. (2000). Using the curriculum vitae to study the career paths of scientists and engineers: an exploratory assessment. Scientometrics, Vol. 49, pp. 419–442.
7. Cañibano, C. et al. (2008). Measuring and assessing researcher mobility from CV analysis: the case of the Ramón y Cajal programme in Spain. Research Evaluation, Vol. 17, pp. 17–31.
8. Fontes, M. (2007). Scientific mobility policies: how Portuguese scientists envisage the return home. Science and Public Policy, Vol. 34, pp. 284–298.
9. Luwel, M. (2005). Job advertisements as an indicator for mobility of researchers: Naturejobs as a case study. Research Evaluation, Vol. 14, pp. 80–92.
10. Laudel, G. (2003). Studying the brain drain: can bibliometric methods help?. Scientometrics, Vol. 57, pp. 215–237.
11. Laudel, G. (2005). Migration currents among the scientific elite. Minerva, Vol. 43, pp. 377–395.
12. Ioannidis, J.P.A. (2004). Global estimates of high-level brain drain and deficit. The FASEB Journal, Vol. 18, pp. 936–939.
13. Hunter, R.S. et al. (2009). The elite brain drain. The Economic Journal, Vol. 119, pp. F231–F251.
14. Qiu, J. (2008) “Scientific publishing: Identity crisis” Nature 451 pp. 766-767.
15. Jonkers, K. (2009). Emerging ties: factors underlying China’s co-publication patterns with Western European and North American research systems in three molecular life science subfields. Scientometrics Vol. 80, pp. 775–795.
16. Crawford, E. Shinn, T. and Sörlin, S. (1993) The Nationalization and Denationalization of the Sciences: An Introductory Essay, in Crawford, E. Shinn, T. and Sörlin, S. (eds.), Denationalizing Science (Dordrecht: Kluwer).

The purpose of this event was to foster open discussion and mutual learning between government officials responsible for shaping and funding the local scientific activities, and the researchers in academia whom they evaluate. To meet these aims, the day was designed to provide international and local perspectives on research evaluation measurements and metrics, and to learn from specific case studies how these methodologies have informed scientific policy and funding in different countries. The meeting focused on three major themes: Theoretical Frameworks and Perspectives; Research Policy on a National Level; and Research Evaluation in Practice.

The first session explored theoretical frameworks and featured Drs Henry Small, Henk Moed and Professor Bar-Ilan, each of whom looked at different ways of using bibliometric data to evaluate scientific output and scientists, study emerging scientific trends and map the evolution of scientific communities. The general conclusion of this session, which was moderated by Professor Pertiz, was that one must first clearly define the objectives and motivations for evaluation and trending studies — only then can one select the appropriate methodology to carry them out. Once the methodology is agreed upon, bibliometric data must be carefully analyzed and scrutinized before any conclusions regarding productivity and output can be made.

The second session, moderated by Professor Moshe Yitzhaki, focused on research evaluation at a national level. Dr. Marc Luwel described how OECD states developed indicators for performance-based funding for basic research in Belgium. Dr. Meir Zadok addressed the history of the development of strict indicators for productivity and impact that are necessary in Israel’s highly competitive scientific research environment. Finally, Dr. Giovanni Abramo reported on the Observatory on Public Research (ORP) system in which national-scale research assessment is based on individual evaluations. This session provided a snapshot of state-level views on the value of scientific output measurements and how appropriate methodologies have been developed to answer local questions and conditions. The task of evaluating research on a state level is not an easy one and certainly not one that a single metric can capture. Although there is always an understandable attempt to have a single and straightforward numeric score that can provide decision makers with a simple way to evaluate research and make funding decisions, the lessons from the different paths taken by different government bodies suggest that a successful process must include high-level decisions on what is being measured, and why; a careful choice of datasets; and rigorous analytics that capture the multifaceted aspects of scientific data.

The third session of the day focused on specific case studies that demonstrated how advanced tools have been used in research evaluation in both academic and government institutions, and was moderated by Professor Benjamin Ehrenberg. In the first part of a joint presentation Mr. Shlomo Herskovic described the national database of R&D statistics and indicators that has been established by Israel’s National Council for Research and Development, primarily in conjunction with the Central Bureau of Statistics and the Neaman Institute for National Policy Research, and discussed its advantages and inherent limitations. In the second part of the presentation Dr. Daphne Getz described work done at the Samuel Neaman Institute in developing an infrastructure of data and knowledge to enable an ongoing analysis of Israeli R&D output, expressed by scientific publications and patents. Mr. Neal Katz demonstrated how research evaluation tools, such as the ones included in Elsevier’s SciVal Suite, are being used to support a variety of strategic initiatives in government and academia. The case studies included work carried out for the UK’s Department for Business Innovation & Skills, the Higher Education Funding Council, Tohoku University (Japan) and the National University of Mexico.

The meeting’s mixture of academic and government perspectives opened up opportunities to evaluate and expand on current research evaluation metrics. While the advanced computation tools that now exist combined with the availability of diverse data types makes the measurement of research output and reaching funding decisions more complex, they nonetheless offer a more rounded and extensive set of practices to be adopted by funding bodies and policy makers. This event captured the fact that research evaluation methodologies are not only dependent on computational power or on datasets, but also must fit with the country’s overall scientific strengths and approach. Future events such as this will take place around the world in 2012 to encourage discussion between government and academia, and open debate on current and future evaluation metrics that will be appropriate for governmental scientific policies and academic capabilities.

The strongest predictor of a journal’s F1000 score is simply the number of article evaluations submitted by F1000 faculty reviewers. Irrespective of their reviewer scores, the number of article evaluations can explain more than 91% of the variation in FFJs (R²=0.91; R=0.96). In contrast, the Impact Factor of the journal can only explain 32% of FFJ variation (R2=0.32; R=0.57).

The rankings of journals based on F1000 scores also reveals a strong bias against larger journals, as well as a bias against journals that have marginal disciplinary overlap with the biosciences.

Larger journals, represented by bigger circles in Figure 1, consistently rank lower than smaller journals receiving the same number of article evaluations. This is most apparent in the “inverted ice-cream cone” shapes in the lower left quadrant of the graph. As I argued previously [1], the method of calculating the F1000 Journal Factor makes it sensitive to enthusiastic reviewers of small journals. This method placed the Journal of Sex and Marital Therapy, which received 12 reviews for its 24 articles in 2010 far above Physical Review Letters, which received just 3 reviews for 3,099 articles.

Phil Davis.
Ithaca NY.

References

Before bibliometrics became widespread the evaluation of science was mainly performed through peer review. This more traditional approach was given a new breath of life when the Faculty of 1000 (F1000) was launched in 2002 to evaluate the quality of biomedical scientific articles based on the opinion of scientific experts. Initially, the papers were evaluated by 1,000 international Faculty members; now F1000 boasts more than 10,000 evaluators spread across 44 subject-specific Faculties. It is worth noting, however, that not all of the Faculty members are active or active to the same extent: Research Trends randomly checked the F1000 records of 20 members of the Reproductive Endocrinology Faculty, and although 4 have contributed more than 10 reviews, half have contributed only 1 or 2 evaluations yet, and a quarter have not made any recommendation yet. Jane Hunter, Managing Director at F1000, commented:

Jane Hunter

“Unsurprisingly, some Faculty Members (FMs) are more active than others and activity levels vary also depending on what other obligations FMs have on a month-by-month basis. Evaluation submission rates drop during congresses and rise immediately after (also unsurprising). Our most productive FMs select and evaluate 10–20 papers a year; our least productive may pick 1 or 2. F1000 has selected and evaluated 91,000 articles to date and these articles have attracted nearly 116,000 evaluations.”

On average, 1,500 new articles are reviewed every month, which according to the F1000 website corresponds to about 2% of all published articles in the biological and medical sciences. This has led to some criticism of the journal rankings derived from the reviews, as overall they are based on very limited coverage of journal content^1,2.For instance, according to Phil Davis, independent researcher and frequent blogger at the Scholarly Kitchen: “because of its limited scope of coverage, the real value of F1000 is not what the aggregate data can tell us about individual journals, but in what experts can tell us about individual articles."  Looking at the 2010 provisional journal rankings, only 5 titles had more than 50% of their papers evaluated, less than 8% of journals had more than 10% of their articles reviewed, and more than 85% had less than 5% of their papers evaluated. According to Jane Hunter, however, this is not a problematic issue:

“F1000’s purpose is to select and evaluate only the top papers in biology and medicine, so it follows that a relatively small percentage of papers from most journals will be included in our database. F1000’s whole system is based on selectivity. This doesn’t invalidate our Journal Rankings. Journals that publish relatively few papers judged as ‘top’ by our Faculty will have a lower FFj (F1000 Journal Factor) in our system and journals that publish a lot of top papers will have a higher FFj.”

One of the unique aspects of the system at the time of the launch was that the ratings were not based on bibliometric data at the journal level, but on expert evaluation at level of individual articles. However, in 2011, in what has been labeled “a 180-degree turn”1 F1000 started a new journal ranking system, including global journal rankings as well as rankings by subject area.

How does it compare to citations?

Citations are usually accepted as a measure of intellectual debt, and although there are negative citations the vast majority of citations are neutral or positive. This can be seen as roughly similar to the F1000 system, in which Faculty members can assign papers to one of three positive quality levels: Exceptional, Must Read, and Recommended. (Interestingly, there is no option to submit negative recommendations.)
However, the similarity ends here: while citations are relatively easy to make (scientific papers routinely include dozens of references), reviews are more time-consuming to produce, and are therefore less numerous. Consequently, it can be argued that F1000 reviews have more weight (there are fewer of them) but also more bias (they can only be positive). However, Jane Hunter disagrees that the absence of negative evaluations introduces bias to the system:

“Negative reviews are simply not what we do. F1000 is a guide to what’s best in science, not a thumbs up/thumbs down review service. There are plenty of comprehensive subject-area reviews published by other companies and we don’t think the world needs another one from us. The fact that we only publish positive reviews doesn’t introduce bias into our system — it is our system. Our subscribers rely on us to tell them what they need to read and not what they need to avoid, so we will never publish negative evaluations. That said, we do publish dissents; if one of our FMs disagrees with another’s article selection or with some aspect of an evaluation, he or she can submit a dissenting opinion, which is then published alongside the article’s evaluation/s on our site. And we also allow registered subscribers to comment on evaluations or dissents, so if they have something to add we invite and encourage them to do so.”

How does it work?

The F1000 Article Factor (FFa) can be calculated from one or several reviews, depending how many are available. If there are several recommendations for one article, the FFa is calculated from the highest rating, which bears a value of 10 for Exceptional, 8 for Must Read, and 6 for Recommended. An incremental value is then added for each of the other ratings (3 for Exceptional, 2 for Must Read, 1 for Recommended). Research Trends was unable to find publicly available explanations for this methodology, and found it difficult to understand why these particular weights were chosen for initial and incremental values, but Jane Hunter was happy to explain:

“The values we assigned to our Recommended, Must Read and Exceptional ratings (6, 8 and 10) are arbitrary, but in essence reflect above-average scores on a 1–10 scale. The rationale for our calculation of total FFa for articles evaluated more than once is also arbitrary — and utilitarian — it made sense to us and seems to work.”

This methodology however raises some concerns about the consistency of the FFa metrics – see example in text box. Furthermore, the FFa calculation gives more weight to the first highest rating and less weight to the following ratings, which has implications for the F1000 Journal Factor (FFj) derived from the FFas: more influence is given to articles with one recommendation compared to articles with several evaluations. As a consequence the FFj appears to be sensitive to enthusiastic reviewers rating numerous papers in small journals.1

Jane Hunter acknowledged this fact, but countered:

“This is not related to our weighting in favor of the highest score a paper receives from us or because we bias our system in favor of number of articles selected over number of evaluations (though we do, intentionally). It’s because at the very specialist end of the scale where there are few journals and we have selected relatively few papers, a small number of additional reviews from a single journal can have a disproportionate impact on a journal’s rank […] For future reference, we will be highlighting articles that have a declared competing interest on our main rankings journal pages in an upgrade planned for later this year. One important feature that sets us apart is complete transparency; our subscribers can easily see how each paper in F1000 was judged, by named experts, and review their reasoning. If there is a competing interest, it is clearly stated.“

Consistency issue: let’s look at some examples

Article A with two Exceptional scores would get an FFa of 13 (10 for the first Exceptional score + 3 for the second Exceptional score). Article B with three Must Read scores and one Recommended score would also get an FFa of 13 (8 for the first Must Read score, 2 for each of the other two Must Read scores, and 1 for the Recommended score), and so would article C with 8 Recommended scores (6 for the first Recommended score + 1 (×7) for the other Recommended scores).

Article A
Rating	Exc	Exc							FFa
Score	10	3							13
Article B
Rating	MR	MR	MR	Rec					FFa
Score	8	2	2	1					13
Article C
Rating	Rec	Rec	Rec	Rec	Rec	Rec	Rec	Rec	FFa
Score	6	1	1	1	1	1	1	1	13

So all three articles would get the same FFa of 13. Let’s imagine now that each article receives one supplementary review (highlighted in red in below table), with an Exceptional score. This would result in article A getting an FFa of 16 (10 for the first Exceptional score and 6 (2 × 3) for the other two Exceptional scores, article B getting an FFa of 17 (10 for the Exceptional score + 6 (3 × 2) for the three Must Read scores + 1 for the Recommended score), and article C getting an FFa of 18 (10 for the Exceptional score + 8 (8 × 1) for the Recommended scores).

Article A
Rating	Exc	Exc	Exc							FFa
Score	10	3	3							16
Article B
Rating	Exc	MR	MR	MR	Rec					FFa
Score	10	2	2	2	1					17
Article C
Rating	Exc	Rec	Rec	Rec	Rec	Rec	Rec	Rec	Rec	FFa
Score	10	1	1	1	1	1	1	1	1	18

So while all articles initially had the same FFa, adding one same rating to each article causes differences in their ranking.

The FFj is calculated from the individual article ratings for a given journal, normalized according to the proportion of eligible scientific articles reviewed by the Faculty. The formula is as follows:

FFj = log10{(Sum of Article Factors) × (Normalization Factor) + 1} × 10

For each journal, the FFa scores are added to obtain the Sum of Article Factors. This sum is then normalized by the Normalization Factor, which is the percentage of articles evaluated by Faculty members compared to all scholarly articles published in the journal according to PubMed. Most bibliometrics indicators normalize for journal size using the number of articles published, but FFj’s normalization is different: going back to our previous bibliometrics analogy, it is similar to multiplying the Impact Factor numerator by the percentage of cited papers rather than dividing it by the number of scholarly papers. This means that FFj’s normalization does not actually account for journal size, but for journal coverage by F1000. For Jane Hunter, this is not a drawback but a benefit:
“Our normalization factor (number of articles selected by F1000/total number of eligible articles) introduces a variable representing journal coverage — or a journal’s F1000 success rate — into our metric. The multiplier accounts for journal size, but it also rewards journals that have had relatively more articles selected by F1000. This is intentional. We want lots of evaluated papers to have a larger positive per-journal effect than a few very highly regarded ones. We believe publishing a lot of good articles is a more reliable indicator of a journal’s value than its ability to publish the occasional megastar.”
The values produced span over several orders of magnitude, so a log scale is applied, and this number is then multiplied by 10 to increase the readability of the final FFj.

Expert Opinion: Ludo Waltman comments

Research Trends spoke to Doctor Ludo Waltman, Bibliometrics Researcher at the Centre for Science and Technology Study at the University of Leiden, about the FFj’s calculation:

“It seems that the developers of the F1000 system wanted to reduce the effect a single publication can have on the overall score of a journal. I guess this is why incremental recommendations have less weight than the initial recommendation. I understand this objective of avoiding 'outliers', but I think there are better ways to achieve this. For instance, the distinction between the initial recommendation and incremental recommendations could be abandoned, giving equal weight to all recommendations of the same type (e.g., all exceptional recommendations have a value of 10, including the incremental ones). To avoid outliers, the final score obtained by adding together the scores obtained from all recommendations a publication has received could be transformed — for instance, by using a square root or logarithmic function. This would also reduce the effect of a single publication with a lot of recommendations, but it has the advantage that consistency of the measurements is maintained. I also have some doubts about the normalization factor used in the calculation of the journal indicators. For instance, suppose we have two journals that each have 100 publications, and in each 50 publications have a single exceptional recommendation and 50 publications do not have any recommendation. This yields a journal score of (50 × 10) × (50%) = 250 for each of the two journals. (For simplicity, I skip the logarithmic transformation performed at the end of the calculations.) Suppose that the two journals are now merged. We then have a single journal with 200 publications, half of them with a single exceptional recommendation and half of them without recommendations. So the score of the merged journal becomes (100 × 10) × (50%) = 500. In other words, journals can increase their score by merging. This means that what is measured by the F1000 journal indicator is first of all the size of a journal (in terms of its number of publications). To obtain a high score, a journal must not only publish high quality articles (i.e., articles that receive recommendations), but it must also publish a large volume of articles. This is different from almost all citation-based journal indicators, such as Impact Factor, SNIP, and SJR (but not Eigenfactor), and most people probably will not be aware of this size-dependence of the F1000 journal indicator.”

What type of rankings does F1000 compute?

Currently, there are three different journal rankings available:

1. Current Journal Rankings: computed on the first day of each month, these are the most up-to-date as they include all evaluations over the previous 12 months, regardless of the publication date of the articles. For instance, February 2012 Current Journal Rankings take into account all recommendations made between 1 February 2011 and 30 January 2012.
2. Provisional Annual Journal Rankings: calculated at the beginning of July, these are based on ratings of articles published in the preceding full calendar year. For instance, 2010 Provisional Annual Journal Rankings take into account evaluations made in 2010 and the first half of 2011 to articles published in 2010; 15 percent of evaluations are received 3 months after an article is published or later: as this adds an extra 3 months for ratings to accumulate, the disadvantage to articles published later in a year is decreased.
3. Final Annual Journal Rankings: also computed at the beginning of July, these take into account evaluations of articles that were published in the last but one full calendar year, enabling the inclusion of 99 percent of potential evaluations for an article regardless of its publication date within a year. For instance, 2010 Final Annual Journal Rankings take into account evaluations made in 2010, 2011, and the first half of 2012 to articles published in 2010.

How does it compare to traditional bibliometrics indicators?

To see how FFj compares with traditional bibliometrics indicators, Research Trends ran a correlation analysis of 2010 Impact Factors versus 2010 provisional FFj for 768 journals mostly of biomedical scope (see Figure 1), in which the proportion of evaluated papers is denoted by the size of the bubble.

  Figure 1 – comparison of 2010 Impact Factor versus 2010 provisional F1000 Journal Factor. Sources: 2011 Journal Citation Reports (© Thomson Reuters); F1000 2010 journal rankings.

The correlation between the two metrics is rather weak overall (correlation coefficient of 0.54), and unsurprisingly at its weakest where only a small proportion of journal content has been evaluated. Yet this correlation does not systematically increase for journals where a high proportion of content has been reviewed. Some of the most noticeable outliers are also some of the journals with the highest Impact Factors (labeled in Figure 1). The analysis was replicated for EigenFactor (correlation coefficient of 0.55), SJR (correlation coefficient of 0.57), and SNIP (correlation coefficient of 0.51). The results presented similar patterns, indicating that bibliometrics indicators and F1000 journal rankings show a different picture of the research landscape: expert ratings seem to measure an alternative dimension to citations. This may be linked to the skewness of the citation distribution in any given journal.

Jane Hunter was not surprised by the results of the analysis: “We wouldn’t expect F1000’s FFjs to directly correlate with bibliometrics indicators — in fact if they did our rankings would be a lot less interesting […] Our metric is based entirely on positive evaluations of science, paper by paper, by panels of experts who read and select articles based solely on their intrinsic — and subjectively judged — importance. Another basic difference between F1000’s metrics and the Impact Factor is that we exclude reviews […] Because of this, journals like Nature Reviews Drug Discovery […] will rank relatively low on F1000, as will any other journal whose Impact Factor is significantly affected by review articles.”

At article level though, there are more similarities: indeed, Allen et al. found a “strong positive association between expert assessment and impact as measured by number of citations and F1000 rating”. They, however, acknowledged that “despite the significant positive correlations between assessments of importance and citations overall, at the individual paper level the analysis showed that there are exceptions; papers that were highly rated by expert reviewers were not always the most highly cited, and vice versa. Additionally, what was highly rated by one set of expert reviewers may not be so by another set; only three of the six ‘landmark’ papers identified by our expert reviewers are currently recommended on the F1000 databases.”³

Where do we go from here?

Jane Hunter offered some concluding remarks:

“We hope that the F1000 Journal Rankings will offer an alternate way of looking at and evaluating scientific success. The strengths and weaknesses of the various ranking systems may balance each other out and ultimately enable scientists to construct a truer picture of where to publish and what to read […] We know there are many ways in which the data generated by F1000 could be used and viewed. Our Article and Journal Factors represent just one way of crunching the individual article ratings allocated by Faculty Members and interpreting the results. The basic data are completely transparent and available on our site, and we’re happy to consider other approaches. The numbers are the numbers, we think they’re interesting, and we know they have other stories to tell.”

Further analyses are needed to help us understand the reasons behind our findings: in particular, it would be very interesting to see how FFjs relate to the distribution of article ratings for each journal. Doing some preliminary research for the article, Research Trends was actually surprised by the apparent lack of studies on the subject, and would therefore like to open a call for papers to the bibliometrics community: we’d love to see more research on F1000 FFa and/or FFj, in particular about their methodologies, or looking at comparison with other metrics. If you’re up for it and would like to publish in Research Trends, just get in touch!

References

Davis, P. F1000 Journal Rankings — The Map Is Not the Territory. Scholarly Kitchen blog post
Butler, D. (2011). Experts question rankings of journals. Nature 478, Vol. 20 doi:10.1038/478020a
Allen, L., Jones, C., Dolby, K., Lynn, D. & Walport, M. (2009). Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS ONE 4, e5910. doi:10.1371/journal.pone.0005910.

Links of interest

• F1000 website http://f1000.com/
• Wikipedia entry http://en.wikipedia.org/wiki/Faculty_of_1000
• Scientist paper http://classic.the-scientist.com/article/display/57586/;jsessionid=0979DD1558B8EA321D99A115FCEECB66

Mapping Scopus

The full set of 2006–2010 citations were extracted from a bibliometric version of the Scopus database as a list of relationships between two publications; and while this includes proceedings and other serials besides journals, I refer solely to journals for clarity. (For more information on bibliometric databases, see the last issue of Research Trends⁶.) The 2006–2010 restriction applies in two ways: not only must the papers citing other papers have been published in this time period, but so too the articles they cite. After excluding journal self-citations, this produced a list of 4,589,565 journal–journal citation relationships, covering 20,213 journals and 27,196,324 citations. (A citation relationship is a count of the citations made from one journal to another within the time period — and so a single citation relationship often represents more than a single citation.)

This full network of data was reduced in order successfully to map it out. Journal–journal citation relationships representing less than 1 percent of the citations made by the citing journal in this set of data were removed, and this resulted in a smaller network containing 19,562 journals (96.8 percent), linked by 377,729 citation relationships (8.2 percent) containing 11,857,165 citations (43.6 percent). These citation relationships were then used to create a network graph, using the Gephi program, in which nodes represent journals, and connecting lines (or edges) the relationships between them.

Gephi is a freeware graphing program⁷ which comes with a range of layout algorithms; the recently-developed ForceAtlas2⁸ was selected as it can quickly position thousands of nodes, and features many properties to refine the graph layout. As with many layout algorithms, ForceAtlas2 is force-directed, which means that unrelated nodes in the network repel one another, while connected nodes attract one another. In this case, the magnitude of these forces was determined by the proportion of citations given by the citing journal to the cited journal out of citations given to all other journals in the network, in the time period 2006–2010. Given the method of reducing the data, edge weights take a value between 0.01 and 1.00, such that the higher the value, the stronger the force of attraction between the two journals. The two forces at work result in a graph which stabilizes over time, until it has reached equilibrium. Figure 1 shows the results of using ForceAtlas2 to lay out our network of 19,562 journals.

Figure 1 – A network of 19,562 journals, linked by 377,729 citation relationships containing 11,857,165 citations mapped using Gephi and the ForceAtlas2 layout algorithm. Each journal is a node (circle) in the map, and edges (lines) between these nodes represent citations from one to the other. Node size is proportional to the total number of citations received by that journal in the time period 2006–2010. Data source: Scopus.

In theory, this map has positioned journals so that related journals are close to one another, and unrelated journals are further apart. But how can this be tested? One option is to use an existing subject classification system, and Figure 2 shows the same map colored according to the subject classifications used by Scopus. There are 27 subject areas, and each is given a different color; journal nodes take the color of the subject area to which they are assigned, but only if they are uniquely assigned to a subject area (journals belonging to multiple subject areas remain gray).

Figure 2 – Each subject area is assigned a color, used to show journals belonging solely to that subject area. Data source: Scopus.

As this labeled map shows, related fields are positioned close to one another; the map can be used to view the position of each subject area in relation to the others — from the health sciences at the left, round the social sciences at the bottom to mathematics at the right, up to physics and chemistry, and round the biological sciences at the top. The most multidisciplinary fields are positioned towards the center of the map, as is clear by the patches of gray journals belonging to multiple fields.

Stuck in the middle with you

Once we have confidence in the layout of the map, we can use it to look at specific subject areas. Figures 3 and 4 show the journals assigned to Environmental Science, and to Physics and Astronomy, respectively. The two subject areas cover a similar area at the right side of the map, stretching almost from the top to the bottom; however, the Physics and Astronomy map shows a much tighter core of journals located at the right edge of the map, while Environmental Science journals do not cluster strongly in any given area. Not only is the subject area multidisciplinary, reaching across the boundaries of other subjects, but the journals within the field are not as closely related to one another as the journals within Physics and Astronomy.

Figure 3 – All journals assigned to Environmental Science. Data source: Scopus.

Figure 4 – All journals assigned to Physics and Astronomy. Data source: Scopus.

Our global map can also be used to look at the crossover between multiple subject areas. Figure 5 again shows Environmental Science journals, and those in Physics and Astronomy, but this time combined with Earth and Planetary Science journals. Each subject area is given a primary color, and secondary and tertiary colors can be used to show the journals assigned to two or all three subject areas.

Figure 5 – Environmental Science journals are colored red; Earth and Planetary Science, yellow; and Physics and Astronomy, blue. Where journals are assigned to two of these subject areas, the secondary colors orange, green and purple are used; where all three, the tertiary color brown. Data source: Scopus.

Using this map to look at the position of the multi-subject journals, we can see that the Earth and Planetary/Environmental journals (in orange) are spread across a much wider area than the Earth and Planetary/Physics and Astronomy journals (in green), which cluster together very tightly. In addition, Earth and Planetary Science journals form a bridge between the other two subject areas.

A map of science formed using the citations indexed by Scopus allows for detailed analysis of not only where a single journal lies in the global map of literature, and the journals to which it is connected, but also the broader subjects that comprise the map, and journal sets that bridge disciplines.

References

Co-citation analysis morphs into patents

The term was coined in 1973 when Henry Small described this methodology in his renowned article ‘Co-citation in the scientific literature: a new measure of the relationship between two documents’, which was cited hundreds of times in the following years. Co-citation analysis has been used for numerous discoveries such as the emergence of author networks in a particular field of investigation, and mapping the emergence of a scientific field^1–3. Small’s technique of using co-citations to analyze a body of research in order to discover scientific trends and collaborations (among other things) was in the most part used in the social sciences; then in the late 1990s the term suddenly started appearing not only in academic discussions but also in patent literature, and the approach has been used by industry pioneers such as Xerox, IBM, Microsoft, and more recently Google.

Technology catches up

How did co-citation analysis, which was conceived as a way to analyze relationships between articles, morph into a product apparatus? Looking at the time frame when patents using the term appear (see Figure 1), it seems that computational development at that time might give us an answer. The late 1990s saw a surge in different types of electronic databases which enabled indexing and retrieval of documents. These years also saw an increase in both co-citation analysis research publications and patents alike, including some interesting surges in 2009 and 2010, when semantic information retrieval and web page rankings took center stage in web-based applications. The first patent that uses co-citation analysis as a method is a Xerox patent that uses co-citation analysis to generate clusters of documents in a database⁴.

Figure 1 – Occurrence of the term ‘co-citation analysis’ in peer reviewed articles versus patents. Sources: TotalPatent and Scopus.

This patent (US Patent 6038574) describes a co-citation analysis method in which hyperlinks in web pages are viewed as references, and the relationships found are used to help create a web page index. Following the Xerox patent application, a series of patents mentioning co-citation analysis in either the method, claims, or references to prior publications appear through the patent literature featuring prominent technological companies (see Figure 2).

Figure 2 – Co-citation patents assignees. Source: TotalPatent.

Looking at some of the patents one can see some interesting applications of co-citation analysis: for example Google Patent (WO2008134373A1), which uses “co-citation analysis, or anchor text analysis (e.g., analysis of text in or near links to the multimedia events) to determine related multimedia events” (Description of Embodiment; 0035). IBM applied co-citation analysis to link analysis on the Web (US7792827B2), and Microsoft used co-citation analysis in their development of latent semantic analysis (US20070239431A1).

Technology? Transferred!

This type of relationship between academic research and product development is a phenomenon that we have learned to expect in areas of medicine or engineering, for example. In these areas, we can more easily find a direct link between theoretical models and their growth into tangible products. What this analysis shows is that social science research can go through a similar evolution, but the cycle through which it develops takes longer to deploy; in this case more than 20 years.

References

In 1974, Alan Ingham and colleagues performed similar rope-pulling experiments in an attempt to verify the existence of the Ringelmann effect². Their findings showed individual performance to decrease significantly with the first few additional co-workers, up to a group of three workers — beyond which additional workers did not significantly decrease individual performance. Ingham et al. described two possible causes for the decrease in individual performance: “It remained unclear whether group members pulled less hard because of incoordination or because of losses in motivation”². Controlling for incoordination in a further experiment in which an individual pulled on the rope — but in some cases believed he was part of a group — the study found a very similar pattern of decreased performance, showing that loss of motivation occurred when co-workers (whether perceived or real) were added to the team.

Ringelmann in research

The Ringelmann effect has real implications for scientific research. Scientific researchers are not simply pulling a rope in unison — but if individual performance decreases as additional researchers come to work on a particular problem, this makes large-scale research projects an unattractive prospect not only for individuals, but also for funding bodies. As Andrea Bonaccorsi and Cinzia Daraio put it, “pressure on public budgets in almost all industrialised countries has led governments to pursue (or at least declare they pursue) efficiency in the allocation and management of resources in the public research sector. The increasing societal demand for accountability and transparency of science also makes it important to demonstrate that public funding follows clear rules”³.

Ton van Raan has investigated the relationship between a variety of bibliometric indicators of size and research quality, at the level of the research group⁴. Van Raan states that “[t]he research group is the most important working floor entity in science, as clearly shown by the internal structure of universities and research institutes”; however, van Raan goes on to say that obtaining data at the level of a research group is far more difficult than for individual authors, institutes or even for whole countries: this is because research groups are not captured in the bibliographic fields attached to papers, such as author names or institutional addresses.

Sizing up subject fields

When Ralph Kenna and Bertrand Berche started to investigate the relationship between the size of a research group and the performance of those groups, they turned to the UK’s Research Assessment Exercise (RAE) as a source of data. The RAE captures data regarding not only the quality of research groups, but their field of research and size. In a rapid chain of papers, Kenna and Berche have compared the sizes of research groups in various fields with the quality assigned to those groups’ research^5–9. Their findings have shown that in every field, there exists a critical mass for increased productivity, with an upper and lower boundary. The lower boundary relates to the classical notion of critical mass, “loosely described as the minimum size a research team must attain for it to be viable in the longer term”; between the lower and upper boundary for critical mass, “the overall strength of research teams tends to rise quadratically with increasing size”; beyond the upper boundary, “research quality levels out”. The basic implication is that “this levelling off refutes arguments which advocate ever increasing concentration of research support into a few large institutions”, and their research shows optimal research group sizes in a number of disciplines⁸. Lately, Kenna and Berche have even used their approach to develop a method of normalizing quality between different research disciplines⁹.

While Kenna and Berche have developed a way to analyze the effect research group size on its overall performance, issues remain regarding how research groups can be assessed. Currently, data supplied for national research assessment programs seem to be the best option, where available; however, tools such as SciVal Strata are starting to address the problem, opening up ways of assessing research groups by citation analysis. (For more information on this approach, the reader is referred to the article by Judith Kamalski and Colby Riese in this issue.)

References

Useful rankings

The authors of the report recognize that rankings are here to stay, given their high level of acceptance by various stakeholders. The report acknowledges the positive aspects of the rankings for universities: they draw the attention of governments to higher education and research; they improve accountability and management methods; and they demonstrate the importance of collecting reliable data. Regarding the robustness of the data on the output, both Web of Science and Scopus were mentioned as reliable databases as far as the sciences and medicine are concerned.

Main findings and criticisms

Going through the comparison of the various methodologies, the report details what is actually measured, how the scores for indicators are measured, and how the final scores are calculated — and therefore what the results actually mean.

The first criticism of university rankings is that they tend to principally measure research activities and not teaching. Moreover, the ‘unintended consequences’ of the rankings are clear, with more and more institutions tending to modify their strategy in order to improve their position in the rankings instead of focusing on their main missions.

For some ranking systems, lack of transparency is a major concern, and the QS World University Ranking in particular was criticized for not being sufficiently transparent.

The report also reveals the subjectivity in the proxies chosen and in the weight attached to each, which leads to composite scores that reflect the ranking provider’s concept of quality (for example, it may be decided that a given indicator may count for 25% or 50% of overall assessment score, yet this choice reflects a subjective assessment of what is important for a high-quality institute). In addition, indicator scores are not absolute but relative measures, which can complicate comparisons of indicator scores. For example, if the indicator is number of students per faculty, what does a score of, say, 23 mean? That there are 23 students per faculty member? Or does it mean that this institute has 23% of the students per faculty compared with institutes with the highest number of students/faculty? Moreover, considering simple counts or relative values is not neutral. As an example, the Academic Ranking of World Universities ranking does not take into consideration the size of the institutions.

Other indicators measuring teaching quality are perceived as strongly questionable, far more so than the ones measuring research. Moreover, the EUA report describes how differences in the way academics publish and cite each other in different fields can create a strong bias in rankings. As such, attempts have been made to normalize across disciplines, and the field-normalization in the Leiden Ranking is a highly regarded example of this effort.

Recommendations

The EUA report makes several recommendations for ranking-makers, including the need to mention what the ranking is for, and for whom it is intended. Among the suggestions to improve the rankings, the following received the greatest attention from the audience:

1. Include non-journal publications properly, including books, which are especially important for social sciences and the arts and humanities;
2. Address language issues (is an abstract available in English, as local language versions are often less visible?);
3. Include more universities: currently the rankings assess only 1–3% of the 17,000 existing universities worldwide;
4. Take into consideration the teaching mission with relevant indicators.

Which ranking, which evaluation tool for which purpose?

Going further, the panel discussion’s participants recommend going beyond the rankings and analysing in detail what information is needed by institutions to assess the diversity of research activities, and to take strategic decisions and implement those choices. Citation analysis or any other single indicator is obviously not sufficient to make decisions on a well-informed basis. The discussion should be to determine what the best tools are depending on the question.

Expectations about the U-Multirank project are high, considering its aim to show the various missions of the universities, far away from a league table, helping students to make choices.

As a conclusion, Jean-Marc Rapp, President of EUA, outlined the next steps in the EUA’s agenda: to analyze both the desired and un-intended consequences of the ranking systems, and comparing the different ways of assessing universities (ranking/rating, benchmarking, quality assessment and so on). This proves again how crucial those evaluation matters are for universities, who are looking for advice about how to make the most informed choices.

Useful links

The original report:  http://www.eua.be/pubs/Global_University_Rankings_and_Their_Impact.pdf

Notes

*The EUA looked at the following rankings: Shanghai Academic Ranking of World Universities (ARWU); Times Higher Education World University Ranking (in cooperation with Quacquarelli Symonds until 2009, and Thomson Reuters from 2010); World’s Best Universities Ranking (QS); Global Universities ranking (Reitor); HEEACT Rankings (Taiwan); EU University-based Research Assessment (AUBR Working Group, EU); Leiden Ranking; CHE University/Excellence rankings; U-Multirank; U-Map classification; and Webometrics.

**Participants involved in the meeting: 1) Presentation of the report: Jan Truszczynski, Director General for Education & Culture, European Commission; Allan Päll, Vice-Chairperson, European Student’s Union; Gero Federkeil, Vice President (VP) International Observatory on Academic Rankings and Excellence. 2) Panel discussion: Chaired by Professor Ellen Hazelkorn, VP Research & Enterprise, Dublin, Ireland; Professor Jean-Pierre Finance,  President Université Henri Poincaré; Sir Howard Newby, Vice-Chancellor, University of Liverpool, United Kingdom; Jens Oddershede, Rector, University of Southern Denmark, Chairman of Universities Denmark.

Curriculum Vitae: Andrejs Rauvargers

Andrejs Rauhvargers was born 1952 in Riga, Latvia, and has a Ph.D. in Chemistry from the University of Latvia. He is Secretary General of the Latvian Rectors’ Conference and Professor at the Faculty of Education at the University of Latvia. He has also served as Deputy State Secretary at Latvian Ministry of Education, where he participated in developing legislation for higher education and was closely involved in the establishment of the higher education quality assurance system in Latvia and its coordination with neighboring countries Estonia and Lithuania. He was also responsible for establishing a system for academic recognition in Latvia.

Internationally, Rauhvargers is a member of the Bologna Follow-Up Group and since 2005 has chaired the working group studying the progress in the 46 ‘Bologna’ countries and preparing the Bologna Stocktaking reports published in 2007 and 2009. He served as president of the European Network of Academic Recognition Centres (ENIC) from 1997 to 2001 and as President of the Intergovernmental Committee of the Lisbon Recognition Convention from 2001 to 2008.

He is the author of several other major reports and a number of publications on various aspects of international higher education, both national and international, and has participated in and managed several higher education reform projects in Croatia, Latvia, Lithuania, Montenegro and Poland. He has been an invited speaker in his field in more than 20 countries. In 2006 Andrejs Rauhvargers was awarded the European Association for International Education’s Constance Meldrum Award for innovation, leadership, and inspiration in international higher education.

Source: http://www.eua.be/about/who-we-are/secretariat/Andrejs-Rauhvargers.aspx

A basic, but important, question relates to who is doing the assessment, and why. Is it a line manager, a funding body, or potential collaborators? The evaluator and the evaluated may even be one and the same, as when a researcher sets out to benchmark their own performance. In these scenarios, the goal might be to recruit, promote, or retain a researcher; to allocate time or equipment to a certain researcher; to determine who should receive awards or money; or to assess one’s own position. Depending on the goal, different aspects of the evaluation of scientific quality may come into play.

In all these cases, one can turn to bibliometric indicators to supplement other methods, such as peer review and interviews. Whether the focus is on productivity, impact, or collaborations, the numbers can shed a light on an individual’s performance. When the researcher in question is established and has published extensively, this should be quite straightforward. But in the early days of a scientific career, this is much harder.

We can distinguish several stages in a researcher’s career. To keep it simple, here we look at three different and somewhat arbitrary career stages: Years 1, 5 and 10.

Year 1: a promising future

Imagine a young and promising researcher in Year 1 of their career, who has not published anything yet — though there’s plenty in the pipeline and lots of exciting ideas. Their time is mostly spent reading and forming ideas for future research. So how can we judge this individual’s performance? Bibliometric analysis is not going to be helpful here, so we might consider looking at their examination results or peer-review comments. Other factors might include their networking activities: are they members of a scientific network? Do they contribute to the network’s discussions? Have they run any workshops or given conferences presentations?

Year 5: underway to the next stage

By now the researcher has published a few articles, and is slowly building a reputation in the field. Their time is mostly spent conducting experiments, networking, and writing up articles. Here, metrics can be more useful in assessing performance than in the earlier stages, but traditional metrics based on averages will not provide an accurate measure because of the small number of publications and citations involved. More immediate metrics could be provided by looking at usage, or downloads, of the researcher’s articles (e.g., how many times have their publications been viewed in a certain database, such as Scopus?). Another relevant aspect could be collaboration: has the researcher published with other reputable researchers in different institutions and countries?

Year 10: established and independent

By the time a researcher gets to this stage of their career, the track record is sufficient for metrics such as the h-index to provide a meaningful measure of output. But one could also look at public presence: how does this person contribute to conferences and events in the relevant subject area, or as a keynote speaker, or in the media? In some fields, patent data may also be relevant.

Let the data do the talking

Research Trends conducted a quick analysis in SciVal Strata (for previous analyses by Research Trends using Strata, see here), a new tool by Elsevier intended to provide a visualization of the activity of a researcher in different stages of their career. Researchers themselves are able to assess their own complete, scientific impact, and also view themselves as part of a team. Strata is not constrained by one particular metric, as a single performance measure is certainly not sufficient to represent an individual researcher’s performance accurately or fairly.

In Figure 1, we see an anonymous young researcher after a few years in science. In Figure 2, we see the experienced, established researcher. While this is just one of many possible ways of looking at performance, it is clear to see how the two profiles differ. The young researcher has only just started to publish, and has not received any citations to date. The established researcher has papers almost every year: the older ones have all been cited, and among the more recent ones there are only a few uncited papers.

Figure 1 – Profile for a young researcher. Only one paper has been published and this is yet to be cited. Source: SciVal Strata, data from Scopus.

Figure 2 – Profile for an established researcher. Dark-colored bars represent cited documents, and lighter-colored bars denote uncited documents.a young researcher. Source: SciVal Strata, data from Scopus.

Take-away lessons for individual assessment

For every stage in a researcher’s career, and for every goal that the person doing the assessment has in mind, there are appropriate tools and measurements. But it is important to bear in mind that consideration must also be given to non-bibliometric indicators that have value for a particular assessor or institution, and which can provide extra information that might tip the balance one way or the other in an overall assessment.

Some quotes by researchers on measuring performance:

“At the start of my scientific career, I measured my performance by looking at the respect I gained from my colleagues. Later, my measure of performance became the public significance of my work as expressed at local and international conferences. And in the final stage of my career, I aim at obtaining results which I can compare with the best achievements in my area of research”

”In my opinion the ‘research performance’ of an individual is very much defined by the influence of his/her ideas over the colleagues in his/her scientific field”