Issue 11 – May 2009

Articles

Breaking boundaries: patterns in interdisciplinary citation

Collaboration has always been an essential aspect of scientific research. Today, technology is making it easier for researchers in one field to access and identify useful research in other subjects. We take a look at citations made to other subjects to see whether collaboration is increasing and in which areas.

Read more >


Science today is separated into many areas that relate to each other in different ways. But are there any areas of research that cross the boundaries of science? Which are the most interdisciplinary areas of research?

This article investigates the major subject areas identified in Scopus that are cited by other subject areas, and attempts to identify those that show the most interdisciplinary citation patterns. We have taken articles published in each subject area between the years 1996–2000 and 2003–2007 and measured citations to these from other subject areas within the same two periods. We can then compare the percentage of citations received by other subjects across both time periods to determine which areas showed the biggest shift in citation patterns.

The results were mixed. For instance, medicine showed very little variation in citation patterns between the two periods, with the majority of citations coming from other medical fields and those in associated life sciences (see Figure 1).

A similar pattern was seen in other medical and life science areas, including biochemistry, neuroscience, nursing, and pharmacology and toxicology. Areas such as arts and humanities, social sciences or psychology also indicated no significant shift in the citation patterns of these fields, although it is worth mentioning that some of these subjects are already diverse by nature.

Figure 1: Differences in citations to medicine from other subject fields.

Figure 1: Differences in citations to medicine from other subject fields.

Branching out…

In contrast, fields such as computer science, engineering, energy and mathematics all showed a great deal of change in the subjects that cite them. Figure 2 illustrates the pattern for mathematics and Figure 3 for computer science.

Figure 2: Differences in citations to mathematics from other subject fields.

Figure 2: Differences in citations to mathematics from other subject fields.

Figure 3: Differences in citations to computer science from other subject fields.

Figure 3: Differences in citations to computer science from other subject fields.

These results indicate a shift in the citation patterns, with different subject areas making citations to academic literature. It also points to a tendency for changes in the nature of the citation relationships of these fields. Indeed, within computer science, shifts of up to 6% are seen in citation activity to other areas, with the main shifts being evident in citations from engineering and mathematics.

To investigate these shifts more closely we compared the top ten most-citing subjects to two fields that seem to show the highest interdisciplinary origin of their citing articles – energy and engineering. Figures 4 and 5 illustrate the percentage breakdown of citations to these areas.

Both energy and engineering have a diverse citation spread and have shown an increase in the “other” areas that have cited them between the two time periods. Energy has shown a 2% shift in citations from “other” fields, while engineering has shown a 6% shift.

Figures 4: Comparison of top ten subjects citing the field of energy, 1996–2000.

Figures 4: Comparison of top ten subjects citing the field of energy, 1996–2000.

Figures 5: Comparison of top ten subjects citing the field of energy, 2003–2007.

Figures 5: Comparison of top ten subjects citing the field of energy, 2003–2007.

Figures 6: Comparison of top ten subjects citing the field of engineering, 1996–2000.

Figures 6: Comparison of top ten subjects citing the field of engineering, 1996–2000.

Figures 7: Comparison of top ten subjects citing the field of engineering, 2003–2007.

Figures 7: Comparison of top ten subjects citing the field of engineering, 2003–2007.

…or converging?

Moshe Kam, Fellow of the Institute of Electrical and Electronics Engineers (IEEE) and Professor at Drexel University, the US, is not surprised by these findings. He says that many research areas that were relatively “isolated” in the past have been developing a stronger interface with disciplines within engineering and computing.

Kam explains: “Rather than interpreting the data as showing increased cross-disciplinary activity, the data may actually indicate that some disciplines and sub-disciplines are converging, or even merging. One example is the increase in the volume of work at the interface of life sciences, computer science, computer engineering and electrical engineering. It is clear from reading papers at this intersection of subjects that many scientists and engineers who were educated in a traditional ‘standalone’ discipline have educated themselves quite well in other areas. At times it is hard to distinguish between the pattern-recognition specialist, the biological-computation expert and the software engineer. There is much less compartmentalization and much more sharing – not only in the results of tasks divided between researchers, but in actually doing the detailed research work together.”

It thus appears that for researchers in certain subjects, the results of research in certain other, complementary fields, are not only of added value; they are becoming essential. If Moshe is correct, the trend is towards convergence rather than cross-disciplinarity for fields that share common research questions and approaches. It remains to be seen whether this will lead to new areas of study at the intersections of complementary fields or greater collaboration between experts within those fields.

Useful links:

IEEE

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
Research Trends Image

Measuring up: how does the h-index correlate with peer assessments?

There are two broad approaches to assess research performance: peer review and the numerous indices based on bibliometric data and analysis. But do they both provide comparable results, and how should they be used? We ask Lutz Bornmann and Hans-Dieter Daniel how the h-index performs against peer review.

Read more >


Since it was first proposed in 2005, Hirsch’s h-index (1) has made a considerable impact on both bibliometricians and the wider scientific community by offering an additional yardstick for assessing individual researchers’ scholarly output and influence. Hirsch’s original paper has been cited more than 280 times in journals, conference proceedings and book series in 14 languages from fields as diverse as medicine and mathematics to engineering and economics (data from Scopus).

The h-index is defined as the number of an individual researcher’s articles that have received the same number (or more) of citations since publication. It is easily derived from any comprehensive list of an author’s papers by ranking them in descending order of citations received and then identifying the rank position at which the number of citations is not less than the ranked value. Since it combines measures of productivity (the upper limit of the h-index for a given author is the total number of papers published) and a proxy for quality (citations received), it has become an attractive all-in-one metric for comparing researchers.

The h-index, and the numerous variants that have proliferated since 2005, can only be used to compare researchers within the same research field; this is true of all metrics that do not account for the publication and citation practices of the various research fields.

Is the h-index a match for peer assessment?

Lutz Bornmann

Lutz Bornmann

An important and interesting question when evaluating individuals is how well the results of bibliometric assessment compare with peer assessment.

For many years, Lutz Bornmann and Hans-Dieter Daniel, at the Swiss Federal Institute of Technology in Zurich and the University of Zurich respectively, have been investigating the review processes used by funding institutions.

Hans-Dieter Daniel

Hans-Dieter Daniel

Explaining their findings, Bornmann says: “In two investigations (3, 4), we have shown that for individual scientists the h-index correlates well with the number of publications and the number of citations that these publications have attracted. This is hardly surprising given that the h-index was proposed to do exactly that.”

In three studies (2, 3, 4), they also examined the relationship between the h-index and peer judgments of research performance. “In these studies, we have shown that the average h-index values of accepted applicants for biomedicine research grants are statistically significantly higher than for rejected applicants.”

Impact versus quantity

Best-practice: getting the most out of the h-index and variants

Use several indicators to measure research performance: the publication set of a scientist, journal, research group or scientific facility should always be described using a multitude of indicators, such as the numbers of publications with zero citations, highly-cited papers and papers for which the scientist is first or last author. Non-publication indicators, such as awards, grant funding and speaking engagements could also be used.

To measure the quality of scientific output using h-index variants, it is sufficient to use just two variants: one that measures productivity and one that measures impact (e.g. the h-index and a-index) (5).

If the h-index is used to evaluate research performance, the fact that it is dependent upon the length of an academic career and the field of study in which the papers are published and cited should always be taken into account. The index should only be used to compare researchers of a similar age and within the same field of study.

However, the h-index has certain disadvantages, including a bias towards older researchers and a failure to place emphasis on highly cited papers. This has led to the development of numerous variants of the h-index. The m-quotient, for example, is computed by dividing the h-index by the number of years that the scientist has been active since the first published paper. Unlike the h-index, the m-quotient avoids a bias towards more senior scientists with longer careers and more publications.

Another variant, the a-index, indicates the average number of citations of publications in the Hirsch core (publications with ≥h citations). In contrast to the h-index, which corresponds to the number of citations for the publication with the fewest citations in the Hirsch core, the a-index is meant to give more weight to highly cited papers.

Bornmann says: “The results of our study (5) show that the h-index and its variants are, in effect, two types of indices: one type describes the most productive core of a scientist’s output and the number of papers in that core; the other type depicts the impact of those papers in the core.”

Using indices wisely

Bornmann and Daniel believe that while their studies (2, 3, 4) provide an initial confirmation of the h-index’s validity, more time and research is required before it can be used in practice to assess scientific work.

“As a basic principle, it is always prudent to use several indicators to measure research performance,” says Bornmann. “The publication set of a scientist, journal, research group or scientific facility should always be described using a multitude of indicators, such as the numbers of publications with zero citations, highly-cited papers and papers for which the scientist is first or last author.”

Bibliometric indicators can and should be used to support peer review, especially where efficiencies are sought. Current research clearly supports the hypothesis that such indicators can approximate the results of peer review, and many research institutes and research councils are already using indices to support their assessments. Informed peer review currently is the state of the art of research evaluation.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Turning the ranking tables on their head: how to improve your standing

Rankings are a useful way for outsiders to assess the relative value of different universities, and administrators are quickly learning that improving their ranking is a useful mark of quality. But what are the rankers looking at, and how can a university show its best side? We speak to one university that is making the rankings work in its favor.

Read more >


In February 2009, the third International Symposium on University Rankings was held in Leiden, the Netherlands. University rankings were discussed from several perspectives: from the position of the researcher or organization developing the rankings to that of the university dean or provost using the rankings to improve their university’s position.

Professor Anthony F.J. van Raan from the Centre for Science and Technology Studies, Leiden University, gave a presentation on the methods used by the various university-ranking systems around the world. For instance, where The Times Higher Education Supplement (THES) bases its analysis on 20% bibliometric input, Shanghai uses 80% and Leiden 100%.

National rankings often also take external inputs, such as average rents for student accommodation in the relevant city, into account. Gero Federkeil, from the Centre for Higher Education Development, explained that some rankings are even bringing their successful alumni into the picture in much the same way that the research community looks at Nobel Prize Laureates. Having a high number of graduates go on to become CEOs at major companies can also be an indicator of quality.

What do these rankings mean to a university?

In many of the discussions, the speakers said that rankings should not be used for resource allocation. It would be wonderful if they could be used to predict, navigate and forecast, but this is not yet possible. This is an area where further research and development are needed.

Professor Luke Georghiou, University of Manchester, explained that while universities do try to improve their ranking, it is less clear how the rankings actually influence behavior.

Climbing up the rankings

One country that has steadily increased its output and quality of papers in recent years is Finland (see Figures 1 and 2). University administrators are very interested to learn how this remarkable success has been achieved.

Figure 1: Article output in Finland has been rising steadily for some years.

Figure 1: Article output in Finland has been rising steadily for some years.

Figure 2: The average h-index of authors in the country went up by 60% in just five years.

Figure 2: The average h-index of authors in the country went up by 60% in just five years.

Jamo Saarti, Library Director at Kuopio University, Finland, says his university has improved its ranking by focusing on strategic research and supporting this with funding. “Kuopio University has made publishing papers in international and high-quality journals a clear priority, and we have been using bibliometric tools to find out where to publish.”

Indeed, analysis of recent articles from the university show that well-cited papers have been published in journals such as Annals of Internal Medicine, Cell, Nature, Nature Genetics, The Lancet and Proceedings of the National Academy of Sciences of the United States of America.

“The management at Kuopio University has used ranking lists as tools in evaluation and we in the library have been very active in acquiring the best possible e-journal collections and promoting the use of these to our researchers,” explains Saarti.

He believes that this focus on high-quality publications, coupled with international collaboration, which has been adopted throughout the university, particularly within the natural sciences and (bio)health sciences, has been key to their success. Figure 3 supports this view, showing that citation levels for the university have been steadily growing.

Figure 3: Kuopio University is succeeding in its goal to increase citations.

Figure 3: Kuopio University is succeeding in its goal to increase citations.

Looking at the rate of citations per subject further supports this approach. Kuopio University’s extra focus on fields such as biological sciences and medicine has paid off, as these were among the university’s top-cited subjects in 2006 and 2007 (see Figure 4).

Figure 4: Kuopio University’s focus on sciences has pushed its citations in these areas to new highs.

Figure 4: Kuopio University’s focus on sciences has pushed its citations in these areas to new highs. Data is field-weighted to eliminate differences in underlying citation activity between disciplines.

Tried and tested

The combination of the university’s strategy, research focus, collaboration with library services and utilization of metrics to track progress provides a very sensible approach to institute management and one that is likely to reap benefits. Indeed, many of the efforts described by Saarti are recognized as key strategies for universities to push forward their research productivity and quality.

Useful links:

International Symposium on University Rankings

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Promoting innovation in Italy

The European paradox, whereby Europe trails the United States in its ability to transfer academic knowledge to industry, is mirrored in Italy, which is falling behind the major European countries. We speak to Giovanni Abramo and Ciriaco Andrea D’Angelo, who believe that bibliometrics could be part of the solution.

Read more >


In 2000, the Lisbon Agenda, which aims to increase European competitiveness, identified numerous areas for improvement. One of its key recommendations was that governments should invest in public research as a source of innovation for industry.

Nine years on, the question remains: has this goal been achieved? While European governments have increased funding, are they also getting return on this investment?

Ciriaco Andrea D’Angelo

Ciriaco Andrea D’Angelo

For Giovanni Abramo of the Italian National Research Council and Ciriaco Andrea D’Angelo, both based at the University of Rome “Tor Vergata”, this has not been the case in Italy. “Our ability to transfer knowledge to industry is very weak. Where Europe lags behind the US, Italy lags behind the major European countries.”

Knowledge transfer is Abramo’s area of expertise. Together with D’Angelo, Abramo he has built a database that contains information on the research output of all Italian researchers. A free query on any subject of interest returns a ranking of relevant experts, based on their productivity and quality of output. This effort led them to bibliometrics and research assessment.

Giovanni Abramo

Giovanni Abramo

Meanwhile, in 2001–2003, Italy launched its first research-evaluation exercise (the triennial VTR), which assesses a sample of an institution’s research for ranking and funding purposes. The VTR is entirely based on peer assessment. Abramo and D’Angelo used this exercise as the springboard for their research, investigating whether bibliometrics can deliver results that are comparable to peer assessment and, if so, whether it could be used to support peer review in general (1).

Towards better and more complete assessment

Abramo and D’Angelo particularly wanted to explore whether bibliometrics could address some of peer assessment’s limitations. Abramo says: “Two major shortcomings of peer reviews are, first, that they can only be carried out on a sample of an institution’s research output. This means it cannot measure productivity. Second, it relies on research institutes being able to select their own best outputs.”

To begin, they tested for correlations between the results of the VTR and a bibliometric analysis of the same data set. “We started by assessing all papers submitted for review and then compared the quality rankings – both methods gave the same results. This means that bibliometrics can be used to support peer review when assessing the hard sciences, thus avoiding peer review’s shortcomings, while also offering the advantages of time and cost efficiencies.”

Abramo also says that relying on universities to select their own best work is a dangerous practice: according to his research, many universities are actually inaccurate when selecting their ‘best’ publications for review. Taking their bibliometric analysis of Italian research output as their starting point, they found that some areas were particularly bad, with certain universities submitting publications whose level of quality fell far below the median of their portfolio of products. In the area of mathematics, for instance, around a quarter of submitted papers had a quality ranking below the median. “This suggests that the universities themselves cannot assess their own value. And if the national assessment is based on what they submit for review, this means the national assessment is meaningless,” he adds.

He suggests that bibliometrics could help at both the selection and submission level within the university – helping them identify their best work – and at the national assessment level. In this way, bibliometric data can help both at the beginning and the end of any research-assessment exercise.

Abramo and D’Angelo hope that Italy will move towards more metric-based assessment in the future. They believe that it is the only way to help Italy improve its ability to allocate scarce public resources more efficiently. “It is also important to consider the transfer of knowledge to government, not just to industry. Our policymakers should be using the output of research that they are actually funding,” says Abramo.

Encouraging collaboration

In another paper, University-industry collaboration in Italy: A bibliometric examination, Abramo and D’Angelo explored where collaborations between universities and industry most frequently occur and how collaboration with industry affects a researcher’s reputation (2).

They discovered that in terms of sheer numbers, most collaboration occurs in the fields of medicine and chemistry. However, the highest concentrations of university-industry co-authored papers are found in the areas of information technology and engineering. This reflects the industries that Italy is strong in,” explains Abramo.

More interesting was their analysis of whether collaboration positively affects quality of output. Their research suggests that it does when academics collaborate with colleagues of other universities or public research institutions, but not when industrial partners are involved. They also studied the motivations for university-industry collaborations. Where industry is seeking new applications and patents, the universities want to publish research results. However, prestigious journals are less inclined to publish this kind of applied research. This means academics have to forgo high-impact publications. So what is in it for them?

According to Abramo: “The incentive for universities is simple: they need the cash to fund research. For academics, it is a tradeoff: they get their funding, but for less prestigious research. They can then do more of the kind of basic research that gets published in high-impact journals.”

In a subsequent investigation, Abramo and D’Angelo found that the way companies select university partners is far from efficient. Even considering the effect of geographic proximity, in 65% of cases, companies could have selected an academic partner closer and with superior scientific performance than the one actually chosen. The bibliometric database set up by Abramo and D’Angelo can help companies identify the best experts.

Collaboration is key to innovation

Abramo and D’Angelo believe that increasing industry-university collaboration is essential if Italy is to achieve its potential: “I cannot understand why governments are prepared to invest so much in research, only to ignore its results,” says Abramo.

He adds that according to the results of a study they have just completed, bibliometrics cannot only support peer review in assessing research efficiency, it can also help in evaluating how universities perform in collaborations with industry (3).

Abramo believes that increased options are the solution. “For me, the ability to better assess public research institutes on a wide range of criteria means we now have the tools to stimulate much better research and technology transfer efficiency than ever before.”

Useful links:
Osservatorio Ricerca Pubblica Italiana (Interface in Italian; queries in English)
Laboratory for Studies on Research and Technology Transfer, University of Rome “Tor Vergata”
IREG-4 Conference
NCURA Magazine

References:

(1) Abramo, G., D’Angelo, C.A. and Capasecca, A. (2009) “Allocative efficiency in public research funding: Can bibliometrics help?”, Research Policy, Vol. 38, pp. 206–215.
(2) Abramo, G. et al (2009) “University-industry collaboration in Italy: A bibliometric examination”, Technovation, doi:10.1016/j.technovation.2008.11.003
(3) Abramo, G. et al (2009) “Assessing the performance of universities in research collaboration with industry”, working paper available pre-publication in English and Italian at: Laboratory for Studies on Research and Technology Transfer, University of Rome “Tor Vergata”
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

…So many papers outside your field?

Computer science is one field that is displaying great leaps towards multidisciplinarity. We ask a computer scientist why he cited outside his subject area.

Read more >


In this issue of Research Trends, we have analyzed the multidisciplinary nature of research and developments. One area that is becoming more multidisciplinary over time is computer science.

Fionn Murtagh

Fionn Murtagh

A good example is Professor Fionn Murtagh’s recent paper, “The structure of narrative: The case of film scripts” in Pattern Recognition, cited in Nature (1). Murtagh is from the Computer Science Department at the University of London. His paper is clearly multidisciplinary, citing many papers from linguistics. Murtagh adds: “it also strongly cites media arts and digital humanities, mathematics and statistics.”

Following a theme

One of the linguistics papers referred to is a paper by Yves Bestgen (2). Murtagh says: “We cited the Bestgen paper due to its content – readying input data for analysis of discourse (in the case of that particular author) and analysis of the particular narrative form provided by a filmscript (in the case of our paper). But I paid no interest whatsoever to whether this paper was categorized as linguistics or otherwise. The way I work is to pursue themes that I think are (very) important, find supporting data, perform extensive evaluation and write all that up.

“Then, if I am convinced at that point that it is presentable, I start thinking of an appropriate journal. I publish, or have published, regularly in journals that are categorized as computer science, statistics, mathematics, physics, astronomy, geology, and other areas.”

Murtagh goes on to explain that in this instance they chose a computer science journal rather than a linguistics journal because “I always seek the most appropriate journal, irrespective of area. I have published in Pattern Recognition before, my first being in 1984, and it is high on my list of ‘personal best’ journals. I am also mindful of discipline-specific evaluations at national and other levels, which can have career implications. I therefore ensure that I have sufficient publications in any given area when I think this is necessary.”

Breaking boundaries

On the topic of multidisciplinarity in general, he says: “I personally have research interests overlapping many fields. My personal aspiration is to always pursue my interests, irrespective of the labels applied to the fields or journals. I would suggest that the core of what computer science is all about is ‘computational thinking’. This is applicable to all disciplines and beyond – to humanities, and to governance and management too.

“However, I do admit that career structures in particular mitigate strongly against cross-disciplinarity. In universities you are in a particular discipline and your performance in all aspects, including research, is evaluated in accordance with the discipline you are in. No one ever said that life is easy!”

References:

(1) Merali, Z. (2008) “Here’s looking at you, kid”, Nature, issue 453, p. 708.

(2) Bestgen, Y. (1998) “Segmentation markers as trace and signal of discourse structure”, Journal of Pragmatics, issue 29, pp. 753–763.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)

Did you know

Scientometrics: the birth of the science of science

Scientometrics is the science of measuring and analyzing science: the science of science. The term ‘scientometrics’ (in Russian: naukometriya) was coined by VV Nalimov in a 1966 paper (1), and he subsequently used the term as the title for his 1969 book on quantitative methods of research on the evolution of science (2). As with many Russian technical reports published during the Cold War, the book was machine-translated into English by the United States Air Force Foreign Technology Division (now the National Air and Space Intelligence Center) in 1971 for distribution to American scientists.

References
(1) Nalimov, VV (1966) “Kolichestvennye metody issledovaniya protsessa razvitiya nauki” [Quantitative methods of research of scientific evolution], Voprosy filosofii [Philosophy Problems], Vol. 12, pp. 38–47.

(2) Nalimov, VV and Mulchenko ZM (1969) Naukometriya. Izuchenie nauki kak informatsionnogo protsessa [Scientometrics. Study of science as an information process]. Moscow: Nauka Publishers.

13
https://www.researchtrends.com/wp-content/uploads/2011/01/Research_Trends_Issue11.pdf
  • Elsevier has recently launched the International Center for the Study of Research - ICSR - to help create a more transparent approach to research assessment. Its mission is to encourage the examination of research using an array of metrics and a variety of qualitative and quantitive methods.