Since the 1990s, research evaluation has been extended to include measures of the (1) social, (2) cultural, (3) environmental and (4) economic returns from publicly funded research. The best known national evaluation system in the world is without a doubt the UK Research Assessment Exercise (RAE), which has evaluated research in the UK since the 1980s. It is due to be replaced by the Research Excellence Framework (REF) in 2014. The REF defines research impact as “... the social, economic, environmental and/or cultural benefit of research to end users in the wider community regionally, nationally, and/or internationally” (1). In the new REF, research impact on society will not only be quantified, but expert panels will also review narrative evidence in case studies supported by the appropriate indicators (informed peer review).

Scientific impact measurement is carried out using a number of established methods (such as the statistical analysis of bibliometric data), which undergo continual development, and is supported by a dedicated community. Research on societal impact measurement is still in its infancy: so far, it does not have its own community with conferences and journals. Godin and Dore see research on the measurement of societal impact as being at the stage where the measurement of research and development (R&D) was in the early 1960s (2). Even though no robust or reliable methods for measuring societal impact have yet been developed, societal impact is already measured in terms of its budget relevance (or will be in the near future). In the REF, 20% of the evaluation of a research unit for the purpose of allocations will be determined by the societal influence dimension (65% by research output and 15% by environment). This uneven relationship between research and practice is astonishing when one considers how long it has taken for the methods for measuring scientific impact to develop sufficiently to reach the stage of budget-relevant practice.

The lack of an accepted and standardized framework for evaluating societal impact has resulted in the "case studies" approach being preferred, not only by the planned REF, but also in other evaluation contexts (3). Although this method is very labor-intensive and very much a ‘craft activity’ (4), it is currently considered the best method. Other approaches such as the "payback framework" are, however, similarly or even more laborious (5). We have developed an approach which, unlike the case study approach (and others), is relatively simple, can be used in almost every subject area and delivers results regarding societal impact which can be compared between disciplines (6). Our approach to societal impact starts with the actual function of science in society: to generate reliable knowledge. Robert K. Merton, who founded the modern sociology of science, used the term communalism to describe one of the norms of science: that scientific knowledge should be considered "public knowledge" and should be communicated not only to other scientists and students, but also to society at large (7).

That is why a document which we would like to refer to as an assessment report, summarizing the status of the research on a certain subject, represents knowledge which is available for society to access. A summary like this should be couched in generally understandable terms so that readers who are not familiar with the subject area or the scientific discipline can make sense of it. Assessment reports can be seen as part of the secondary literature of science, which has up to now drawn on review journals, monographs, handbooks and textbooks (primary literature is made up of the publications of the original research literature). They would be items of communal knowledge made available to society. To ensure that they are of high quality, they should be written by scientists (specialists in their field) and should undergo peer review to determine their correctness. The reviewers are asked to recommend the publication or rejection of the report and possibly formulate suggestions for improvement to the submitted work (8). Since the report will be read by scientists from other fields and non-scientists, it should be reviewed not only by an expert in the field but also by a stakeholder from the government, the industry or an advice centre.

What is an assessment report?

  • It can be produced for almost every discipline
  • It summarizes the status of research for those outside of the expert community
  • It should be written by scientists (specialists in their field), reflect above all research excellence and undergo peer review to determine its quality
  • It could be narrative reviews reporting on just the research results from the primary literature. For subjects in which effect sizes are available in empirical studies, on the other hand, it would be possible to carry out meta-analyses (the statistical analysis of a large collection of analysis results from individual studies)
  • It should be couched in generally understandable terms so that readers who are not familiar with the subject area or the scientific discipline (e.g. stakeholders from government, industry or an advice centre) can make sense of it
  • It could be produced separately, or be integrated as part of literature reviews written for the scientific community (as sections summing up the situation for those outside of the community)
  • In order to establish the assessment report as a service provided by science for society, it would be important firstly for research funders to make the production of assessment reports obligatory, and secondly that when research is evaluated (by institutes, research groups and individuals) assessment reports are regarded as content for society to generate societal impact
  • Societal impact is given when the content of an assessment report is addressed outside of science (in a government document, for example). This can be verified with tools which measure the attention that academic papers receive online (for example, Altmetric or ImpactStory).

Societal impact of research is obtained when the content of a report is addressed outside of science (in a government document, for example). This can be verified with tools which measure the attention that academic papers receive online. Altmetric, for example, captures hundreds of thousands of tweets, blog posts, news stories and other pieces of content that mention academic documents. In Scopus, the Elsevier literature database, it is possible to display not only the extent of the scientific impact, but also that of the societal impact of individual publications. This type of societal impact measurement would be carried out in a similar way as the measurement of scientific impact. In other words this would mean applying an established and successful method of measuring scientific impact (the analysis of citations in scientific publications) to the measurement of societal impact, which has clear benefits. For tools like Altmetric, citations should ideally be classified and weighted: for example, a citation by a member of the president’s council of economic advisors should have a different weight than a mention in a random blog post.

The assessment reports produced by the Intergovernmental Panel on Climate Change (IPCC)  are a good example of public knowledge which aims to generate societal impact from one subject area. This panel was founded in 1988 by UNEP, the United Nations Environmental Program, and the World Meteorological Organization (WMO) to summarize the status of climate research for political decision-makers. Its main function is to assess the risks and consequences of global warming to the environment and to society and to develop strategies to avoid it. Climate research is exceptional in that it is a strongly interdisciplinary field like almost no other and encompasses many of the Natural Sciences, such as Biological and Environmental Sciences, Atmospheric Chemistry, Meteorology and Geophysics. There are many points at which they intersect with Politics and Economics. With a broad focus in many fields and a rapidly expanding volume of publications, this area of research has become confusing even for insiders; it requires processing and summarizing to allow the results to be used outside of science and make them available for implementation in the form of policies. Working groups involving numerous scientists collate the results of research for the assessment report at regular intervals. The goal is a coherent representation of the research. As the reports reflect the current consensus of science on climate change, they have become the most important basis for scientific discussion and political decisions in this area. The scientific impact of the IPCC reports can be measured by using citations in scientific publications (such as the Web of Science and Scopus literature databases). The societal impact can be quantified with tools such as Altmetric (see above).

Measuring scientific impact with citations in journal papers can be used to great effect in the Physical and Life Sciences but hardly at all in Social Sciences and Arts & Humanities. An assessment report, on the other hand, can be produced for almost every discipline and its societal impact can be clearly measured. Since Social Sciences and Arts & Humanities are disciplines where impact is generally very difficult to measure, assessment reports offer the advantage of reporting not only on journal papers and monographs, but also on exhibitions and art objects. In our view, it is of fundamental importance that an assessment report reflects above all research excellence in the subject area. Thus, only publications which have been previously subjected to peer review should be included in an assessment report. For certain issues or in certain subject areas, it could be helpful if the reports for society were not produced separately, but integrated as part of the literature reviews written for the scientific community. This could be done in sections summing up the situation for those outside of the community (as a sort of comprehensive layman’s summary).

Although in many countries there is a wish (and a will) to measure societal impact, “it is not clear how to evaluate societal quality, especially for basic and strategic research” (9). In many studies in which societal impact has been measured, it is more often postulated than demonstrated by research. With the breadth of subject matter and complex content of the challenges facing society today (such as population growth or environmental pollution), the demand for available information to be summarized and evaluated for social and political purposes is rising. We have presented an approach with which the societal impact of research outcomes can be initiated and measured (6). We suggest that, as with the IPCC, assessment reports are written on certain research subjects which summarize the status of the research for those outside of the expert community. Tools such as Altmetric can verify the extent to which the assessment reports generate an impact. It would be desirable if these tools were to search through documents for citations used in various contexts for decisions, such as documents from governmental bodies, advisory bodies and consumer organizations.

 

References

(1) RQF development advisory group (2006) Research quality framework: assessing the quality and impact of research in Australia. The recommended RQF (report by the RQF development advisory group). Canberra, Australia: Department of Education, Science and Training.
(2) Godin, B., & Dore, C. (2005) Measuring the impacts of science; beyond the economic dimension, INRS Urbanisation, Culture et Société. Paper presented at the HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland. Available at: http://www.csiic.ca/PDF/Godin_Dore_Impacts.pdf
(3) Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, Vol. 13, No. 8, pp. 673-676.
(4) Martin, B. R. (2011) “The Research Excellence Framework and the 'impact agenda': are we creating a Frankenstein monster?”, Research Evaluation, Vol. 20, No. 3, pp. 247-254.
(5) Bornmann, L. (2013) “What is societal impact of research and how can it be assessed? A literature survey”, Journal of the American Society of Information Science and Technology, Vol. 64, No. 2, pp. 217-233.
(6) Bornmann, L., & Marx, W. (in press) “How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons”, Scientometrics.
(7) Merton, R. K. (1973) The sociology of science: theoretical and empirical investigations. Chicago, IL, USA: University of Chicago Press.
(8) Bornmann, L. (2011) “Scientific peer review”, Annual Review of Information Science and Technology, Vol. 45, pp. 199-245.
(9) van der Meulen, B., & Rip, A. (2000) “Evaluation of societal quality of public sector research in the Netherlands”, Research Evaluation, Vol. 9, No. 1, pp. 11-25.
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)