How do we know whether a particular researcher is promising, or to whom funding should be allocated, or who the best candidate for a certain position is? Research Trends tries to offer some guidance.

A basic, but important, question relates to who is doing the assessment, and why. Is it a line manager, a funding body, or potential collaborators? The evaluator and the evaluated may even be one and the same, as when a researcher sets out to benchmark their own performance. In these scenarios, the goal might be to recruit, promote, or retain a researcher; to allocate time or equipment to a certain researcher; to determine who should receive awards or money; or to assess one’s own position. Depending on the goal, different aspects of the evaluation of scientific quality may come into play.

In all these cases, one can turn to bibliometric indicators to supplement other methods, such as peer review and interviews. Whether the focus is on productivity, impact, or collaborations, the numbers can shed a light on an individual’s performance. When the researcher in question is established and has published extensively, this should be quite straightforward. But in the early days of a scientific career, this is much harder.

We can distinguish several stages in a researcher’s career. To keep it simple, here we look at three different and somewhat arbitrary career stages: Years 1, 5 and 10.

Year 1: a promising future

Imagine a young and promising researcher in Year 1 of their career, who has not published anything yet — though there’s plenty in the pipeline and lots of exciting ideas. Their time is mostly spent reading and forming ideas for future research. So how can we judge this individual’s performance? Bibliometric analysis is not going to be helpful here, so we might consider looking at their examination results or peer-review comments. Other factors might include their networking activities: are they members of a scientific network? Do they contribute to the network’s discussions? Have they run any workshops or given conferences presentations?

Year 5: underway to the next stage

By now the researcher has published a few articles, and is slowly building a reputation in the field. Their time is mostly spent conducting experiments, networking, and writing up articles. Here, metrics can be more useful in assessing performance than in the earlier stages, but traditional metrics based on averages will not provide an accurate measure because of the small number of publications and citations involved. More immediate metrics could be provided by looking at usage, or downloads, of the researcher’s articles (e.g., how many times have their publications been viewed in a certain database, such as Scopus?). Another relevant aspect could be collaboration: has the researcher published with other reputable researchers in different institutions and countries?

Year 10: established and independent

By the time a researcher gets to this stage of their career, the track record is sufficient for metrics such as the h-index to provide a meaningful measure of output. But one could also look at public presence: how does this person contribute to conferences and events in the relevant subject area, or as a keynote speaker, or in the media? In some fields, patent data may also be relevant.

Let the data do the talking

Research Trends conducted a quick analysis in SciVal Strata (for previous analyses by Research Trends using Strata, see here), a new tool by Elsevier intended to provide a visualization of the activity of a researcher in different stages of their career. Researchers themselves are able to assess their own complete, scientific impact, and also view themselves as part of a team. Strata is not constrained by one particular metric, as a single performance measure is certainly not sufficient to represent an individual researcher’s performance accurately or fairly.

In Figure 1, we see an anonymous young researcher after a few years in science. In Figure 2, we see the experienced, established researcher. While this is just one of many possible ways of looking at performance, it is clear to see how the two profiles differ. The young researcher has only just started to publish, and has not received any citations to date. The established researcher has papers almost every year: the older ones have all been cited, and among the more recent ones there are only a few uncited papers.

Figure 1 – Profile for a young researcher. Only one paper has been published and this is yet to be cited. Source: SciVal Strata, data from Scopus.

Figure 2 – Profile for an established researcher. Dark-colored bars represent cited documents, and lighter-colored bars denote uncited documents.a young researcher. Source: SciVal Strata, data from Scopus.

Take-away lessons for individual assessment

For every stage in a researcher’s career, and for every goal that the person doing the assessment has in mind, there are appropriate tools and measurements. But it is important to bear in mind that consideration must also be given to non-bibliometric indicators that have value for a particular assessor or institution, and which can provide extra information that might tip the balance one way or the other in an overall assessment.

Some quotes by researchers on measuring performance:

“At the start of my scientific career, I measured my performance by looking at the respect I gained from my colleagues. Later, my measure of performance became the public significance of my work as expressed at local and international conferences. And in the final stage of my career, I aim at obtaining results which I can compare with the best achievements in my area of research”

”In my opinion the ‘research performance’ of an individual is very much defined by the influence of his/her ideas over the colleagues in his/her scientific field”

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)