Level the playing field in scientific international collaboration with the use of a new indicator: Field-Weighted Internationalization Score
Hans Pohl, Guillaume Warnan and Jeroen Baas propose a new indicator to measure collaboration at researcher and institute level.
International relations are and have always been inherent in higher education and research (1). However, internationalization of higher education institutions (HEIs) exhibits a growing trend, as illustrated by bibliometric data (2). Amongst other things, the internationalization trends challenge the leadership of the HEI and lead to changes in management structure (3).
An assessment of the internationalization impact has to be aligned with the core missions of the HEI (4) and there is a need to manage and measure various internationalization aspects:
“Without a clear set of rationales, followed by a set of objectives or policy statements, a plan or set of strategies, and a monitoring and evaluation system, the process of internationalization is often an ad hoc, reactive, and fragmented response to the overwhelming number of new international opportunities available” (5).
Common internationalization indicators include share of international students and staff, and share of international co-publications. Indicators of this type are widely used for comparisons, rankings such as QS World University Rankings and even for the allocation of funding to HEIs (6).
This paper addresses one clearly defined but rather crude indicator: the share of international co-publications (for a given researcher or institution). The indicator has several advantages, among them relatively unbiased data, the possibility to study all levels from individual researchers to countries and the ease of interpreting it. But there are also weaknesses. Comparisons of researchers, groups of researchers or even HEIs with different scientific profiles are difficult, as the typical share of international co-publications varies substantially between different scientific fields. This is illustrated in Figure 1, which also shows how the share of international co-publications has increased over time in all scientific fields.
Figure 1 - Share of international co-publications per scientific field 2009 and 2013. Source: Scopus
Another weakness is that the share of international co-publications changes with different types of publications (see Table 1).
|Scientific field||Share of international co-publications|
|All types||Articles||Conference proceedings||Reviews|
Table 1 - International co-publications per publication type overall and for 3 different scientific fields, 2013
The aim of this paper is to develop and test an indicator that eliminates these weaknesses without losing the advantages. The indicator described in this piece, named the Field-Weighted Internationalization Score (FWIS), builds on the Field-Weighted Citation Impact (FWCI) calculation.
While many articles study the concept of scholarly collaboration (7) or point out the importance of international collaboration (8), the assessment of international collaboration remains a more limited field of study. It really became a subject of interest in the 1990s (9-11). Indexes were created (12), but never aimed at comparing institutions or research entities with one another.
The FWIS is calculated using the same base-normalization as is applied in the calculation of the Field-Weighted Citation Impact (13). This in turn is based on the scientific consensus reached recently (14), after criticism that normalization scores should be calculated at the publication level (15) and the contributing counts need to be fractionalized (16). In essence, this means that each publication will have a calculated expected value, normalized for publication year, document type, and field. The FWCI score for each publication is the actual value divided by the expected value.
The same logic is used for the calculation of the FWIS, and instead of citation counts, a simple binary indication of the presence of collaboration on the publication is included. Citation counts behave a little different, as a publication can be cited for instance twice as much as another publication. The simple binary indication of an international co-publication recognizes only two states: either the publication is internationally co-authored (value is 1), or the publication is not (value is 0). This calculation therefore relates to the percentage of internationally co-authored publications, rather than the average internationality of publications (where FWCI does relate to the average number of citations per publication).
In order to overcome the pitfall of measuring collaboration rates against a global rate – when most entities will appear to achieve collaboration rates that are higher than expected – the expected value of collaboration per publication is calculated by weighting the publications by the number of countries that appear on the publication.
To illustrate the methodology with an example (see Table 2 - we will first assume all documents are from the same year, document type and subject, and gradually add complexity to the example to fully understand the calculations): suppose we have a total of 4 publications in our database, which includes 3 different countries: China, USA and UK. The global share of international co-publications is 50% as 2 out of 4 are internationally co-authored.
Table 2 - First example with 4 publications and 3 countries
In our example, China has 50% international publications, USA has 67% and UK 100%. If you were to compare these percentages to the global average, it would appear as if all of these are above or exactly at the global average. To remedy this effect in collaboration, we multiply the weight of publications by the number of collaborating countries contributing to the publication. In our example, that would mean a global average of (1*0+1*0+2*1+3*1)/(1+1+2+3)=71% and not 50%.
Multiplying by number of countries on a publication means that the percentage of internationally co-authored publications is affected by the average number of countries on a publication. When comparing values that have been calculated for different fields (and thus having different average number of countries per publication) this indirectly causes different results. If for instance in a field, without multiplying by country, a group of researchers have 30% international co-publications vs. a global average of 15% (twice as high), and in another field the same group has 10% international collaboration vs. a global average of 5% (also twice as high), it may be that the FWIS derived from those publications per field is different if the average number of countries on international publications is different. The rationale of this difference is that fields with more countries per publication have a higher likelihood of international collaboration.
FWIS uses a publication-oriented approach, which means that an expected and actual value for each publication is calculated. The expected count is derived by taking the total number of international co-publications divided by the total number of publications, and by weighting these counts with the number of countries involved. This would mean in our example (see Table 3): (1*0+1*0+2*1+3*1)/(1+1+2+3)=0.71. The FWIS for each publication is derived by dividing the actual value (0 or 1) by the expected value.
|Publication||China||USA||UK||Count of countries||International?||Expected score per publication||FWIS|
Table 3 - Addition of the count of countries and FWIS taking into account the number of countries
In order to calculate the score for an entity (entity could for example be a country, institution or group of researchers), we simply take the arithmetic mean of each FWIS score for the entity’s publications. For instance, for China this would be: (0+1.41)/2=0.71.
When calculating the global score for the entire dataset t – as is required to validate the calculation and end up with a score of 1.00 – each publication again needs to be weighted, using the count of countries that are present on the publication. In our example, the global value is derived by: (0*1+0*1+1.41*2+1.41*3)/(1+1+2+3)= 1.00. The same weighting is required when calculating the score for entities that span multiple countries. For instance, for a continent, the value is derived by multiplying the score for each country that is part of that continent.
To fully understand the model, we also need to consider the properties from FWCI that have remained the same: normalization by subject, publication type and year. Normalizing publication types and year of publication are relatively straightforward, by simply taking the average international rate within each subgroup (for example, 2008 and reviews). Subjects are a little more complicated, because publications can belong to multiple subjects at the same time.
Let’s take a look at our initial example, and this time adding subject classifications to the publications (see Table 4).
|Publication||China||USA||UK||Count of countries||Subject classification|
Table 4 - Addition of subject classifications
In order to account for which subject a publication belongs to, and thus to calculate the expected value per subject, each publication is weighted to the subject by fractionalizing the publication. For publication #3, this means that 50% of the publication counts towards subject B and 50% to subject C. For each subject, the expected value is therefore in this example (multiplying by country-count and fractional subjects):
Subject A: (1*1*0 + 3*0.5*1)/(1*1+3*0.5)=0.6
Subject B: (1*1*0 + 2*0.5*1)/(1*1+2*0.5)=0.5
Subject C: (2*0.5*1 + 3*0.5*1)/(2*0.5+3*0.5)=1.0
To form the expected counts using each of these normalized scores, we take the harmonic mean of the subjects per publication (see Table 5). For publication #3 this is 2/((1/0.5)+(1/1))=0.67, and for publication #4 this is 2/((1/0.6)+(1/1))=0.75. The FWIS per publication again is derived by dividing International (1 or 0) by the expected score.
|Publication||China||USA||UK||Count of countries||Subject classification||Inter-national||Expected score per publication||FWIS|
|#3||1||1||2||B, C||1 (yes)||0.67||1.5|
|#4||1||1||1||3||A, C||1 (yes)||0.75||1.33|
Table 5 - Addition of the FWIS per publication taking into account the number of countries and the subject classification
In this example, the FWIS for China is (0+1.33)/2=0.66. To validate the model, the global average still needs to be 1.0 across the subject fields. Applying the same country-count weighting as before, this is calculated as (0*1+0*1+1.5*2+1.33*3)/(1+1+2+3)=1.0.
Testing the new metric
The FWIS was recently tested on a real-case example. In collaboration with the Swedish Foundation for International Cooperation in Research and Higher Education (STINT), we compared 28 Swedish universities on the basis of their level of internationalization.
A first analysis was based on the share of international co-publications (see Figure 2 – right part). That analysis put forward institutions focused on disciplines where international collaboration is naturally strong such as Economics, Econometrics and Finance (Stockholm School of Economics) or Life Sciences (Stockholm University).
A second analysis used the FWIS (see Figure 2 – left part), and both rankings were finally compared (see Figure 3).
Figure 2 - FWIS and share of international co-publications per Swedish university - 2013
40% of the institutions (11 out of 28) experienced a major change (greater than four places) in their ranking position due to the change of indicator used as a basis for the ranking. Source: SciVal and Scopus.
Figure 3 - Comparison of the ranking of Swedish universities based on the share of international co-publications or FWIS - 2013. Source: SciVal and Scopus. Note: This Figure was updated on 21 November 2014 to correct the placement of the blue captions.
The example of Luleå University of Technology is very representative of the impact of the use of the FWIS instead of the share of international co-publications. Luleå focuses predominantly on engineering-type disciplines (see Figure 4) which are typically quite weak in terms of international collaboration (see Figure 1).
Figure 4 - Split of publication output per journal category for Luleå University of Technology - 2013. Source: SciVal.
Luleå’s share of international co-publications in those disciplines may appear limited (around 50%), but they are much greater than the global average (see Table 6). When changing from a ranking based on share of international co-publications to one based on FWIS, Luleå moves up 7 positions.
The FWIS indicator gives Luleå University of Technology a better value as it takes the specific mix of the university’s scientific fields into account, i.e. the output of Luleå is compared fairly with that of peers instead of assuming that all universities have the same mix of scientific production.
Responding to the need for better management and understanding of internationalization of research and higher education, this paper elaborates and tests a new indicator relating to international research collaboration. The proposed FWIS indicator is argued to enhance the possibilities to measure and compare internationalization of HEIs. The very common indicator using the share of international co-publications includes biases due to scientific profile, type of publication and year of publication. Using a method similar to the calculation of FWCI, the proposed indicator eliminates these biases with the same underlying dataset.