In 2000, the Lisbon Agenda, which aims to increase European competitiveness, identified numerous areas for improvement. One of its key recommendations was that governments should invest in public research as a source of innovation for industry.

Nine years on, the question remains: has this goal been achieved? While European governments have increased funding, are they also getting return on this investment?

Ciriaco Andrea D’Angelo

Ciriaco Andrea D’Angelo

For Giovanni Abramo of the Italian National Research Council and Ciriaco Andrea D’Angelo, both based at the University of Rome “Tor Vergata”, this has not been the case in Italy. “Our ability to transfer knowledge to industry is very weak. Where Europe lags behind the US, Italy lags behind the major European countries.”

Knowledge transfer is Abramo’s area of expertise. Together with D’Angelo, Abramo he has built a database that contains information on the research output of all Italian researchers. A free query on any subject of interest returns a ranking of relevant experts, based on their productivity and quality of output. This effort led them to bibliometrics and research assessment.

Giovanni Abramo

Giovanni Abramo

Meanwhile, in 2001–2003, Italy launched its first research-evaluation exercise (the triennial VTR), which assesses a sample of an institution’s research for ranking and funding purposes. The VTR is entirely based on peer assessment. Abramo and D’Angelo used this exercise as the springboard for their research, investigating whether bibliometrics can deliver results that are comparable to peer assessment and, if so, whether it could be used to support peer review in general (1).

Towards better and more complete assessment

Abramo and D’Angelo particularly wanted to explore whether bibliometrics could address some of peer assessment’s limitations. Abramo says: “Two major shortcomings of peer reviews are, first, that they can only be carried out on a sample of an institution’s research output. This means it cannot measure productivity. Second, it relies on research institutes being able to select their own best outputs.”

To begin, they tested for correlations between the results of the VTR and a bibliometric analysis of the same data set. “We started by assessing all papers submitted for review and then compared the quality rankings – both methods gave the same results. This means that bibliometrics can be used to support peer review when assessing the hard sciences, thus avoiding peer review’s shortcomings, while also offering the advantages of time and cost efficiencies.”

Abramo also says that relying on universities to select their own best work is a dangerous practice: according to his research, many universities are actually inaccurate when selecting their ‘best’ publications for review. Taking their bibliometric analysis of Italian research output as their starting point, they found that some areas were particularly bad, with certain universities submitting publications whose level of quality fell far below the median of their portfolio of products. In the area of mathematics, for instance, around a quarter of submitted papers had a quality ranking below the median. “This suggests that the universities themselves cannot assess their own value. And if the national assessment is based on what they submit for review, this means the national assessment is meaningless,” he adds.

He suggests that bibliometrics could help at both the selection and submission level within the university – helping them identify their best work – and at the national assessment level. In this way, bibliometric data can help both at the beginning and the end of any research-assessment exercise.

Abramo and D’Angelo hope that Italy will move towards more metric-based assessment in the future. They believe that it is the only way to help Italy improve its ability to allocate scarce public resources more efficiently. “It is also important to consider the transfer of knowledge to government, not just to industry. Our policymakers should be using the output of research that they are actually funding,” says Abramo.

Encouraging collaboration

In another paper, University-industry collaboration in Italy: A bibliometric examination, Abramo and D’Angelo explored where collaborations between universities and industry most frequently occur and how collaboration with industry affects a researcher’s reputation (2).

They discovered that in terms of sheer numbers, most collaboration occurs in the fields of medicine and chemistry. However, the highest concentrations of university-industry co-authored papers are found in the areas of information technology and engineering. This reflects the industries that Italy is strong in,” explains Abramo.

More interesting was their analysis of whether collaboration positively affects quality of output. Their research suggests that it does when academics collaborate with colleagues of other universities or public research institutions, but not when industrial partners are involved. They also studied the motivations for university-industry collaborations. Where industry is seeking new applications and patents, the universities want to publish research results. However, prestigious journals are less inclined to publish this kind of applied research. This means academics have to forgo high-impact publications. So what is in it for them?

According to Abramo: “The incentive for universities is simple: they need the cash to fund research. For academics, it is a tradeoff: they get their funding, but for less prestigious research. They can then do more of the kind of basic research that gets published in high-impact journals.”

In a subsequent investigation, Abramo and D’Angelo found that the way companies select university partners is far from efficient. Even considering the effect of geographic proximity, in 65% of cases, companies could have selected an academic partner closer and with superior scientific performance than the one actually chosen. The bibliometric database set up by Abramo and D’Angelo can help companies identify the best experts.

Collaboration is key to innovation

Abramo and D’Angelo believe that increasing industry-university collaboration is essential if Italy is to achieve its potential: “I cannot understand why governments are prepared to invest so much in research, only to ignore its results,” says Abramo.

He adds that according to the results of a study they have just completed, bibliometrics cannot only support peer review in assessing research efficiency, it can also help in evaluating how universities perform in collaborations with industry (3).

Abramo believes that increased options are the solution. “For me, the ability to better assess public research institutes on a wide range of criteria means we now have the tools to stimulate much better research and technology transfer efficiency than ever before.”

Useful links:
Osservatorio Ricerca Pubblica Italiana (Interface in Italian; queries in English)
Laboratory for Studies on Research and Technology Transfer, University of Rome “Tor Vergata”
IREG-4 Conference
NCURA Magazine

References:

(1) Abramo, G., D’Angelo, C.A. and Capasecca, A. (2009) “Allocative efficiency in public research funding: Can bibliometrics help?”, Research Policy, Vol. 38, pp. 206–215.
(2) Abramo, G. et al (2009) “University-industry collaboration in Italy: A bibliometric examination”, Technovation, doi:10.1016/j.technovation.2008.11.003
(3) Abramo, G. et al (2009) “Assessing the performance of universities in research collaboration with industry”, working paper available pre-publication in English and Italian at: Laboratory for Studies on Research and Technology Transfer, University of Rome “Tor Vergata”
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)