Research Trends -

Which scientific stories are most shared on social media networks such as Twitter and Facebook?

Research on human health and social issues are often perceived as being the most shared scientific stories on social media networks such as Twitter and Facebook and – given their mainstream appeal – are often suggested to dominate the popular discussion around scholarly research online, but skeptics, such as David Calquhoun argue for their irrelevance: “Scientific works get tweeted about mostly because they have titles that contain buzzwords, not because they represent great science” (1).

So which is it to be? And do articles attracting social media attention also get the attention of scholars and the mass media? In this article, we seek to provide an approach to answering these questions.

With the rise of online scholarly publishing and the concomitant rise in the desire to create indicators of online attention to research articles and related outputs have come a number of providers of article-level data. A leading commercial provider of such data - collectively known as ‘altmetrics’ - is Altmetric.com, which tracks a variety of different indicators in four broad groups: Social Activity (e.g. Tweets and Facebook mentions), Mass Media (e.g. mentions on news sites such as BBC and CNN), Scholarly Commentary (e.g. mentions in scientific blogs), and Scholarly Activity (e.g. articles in reader libraries such as Mendeley). The overall collection and analysis of these references are brought together under the label “altmetrics”.

In terms of the volume of online mentions of scholarly articles, Twitter and other social networks provide by far the largest number of data points. However, given Twitter’s broad user base (the majority being non-academics) and limited information content (being restricted to 140 characters per tweet), other indicators may be more significant in terms of understanding scholarly usage (2). For example, Mendeley and CiteULike are examples of sharing and collaboration platforms used predominantly by researchers, while the mass media and scientific blogs tracked by Altmetric.com are written by professional science journalists or researchers themselves.

Methodology

Data were collected from the Altmetric.com API over four months ending January 17th, 2014. On this date, the latest altmetric indicator data for all papers published in a selection of journals in 2013 with any online mentions captured by Altmetric.com were downloaded for analysis; in total, 13,793 articles with at least one altmetric indicator datapoint were included in this study. Please note, the actual Journals monitored are detailed in the raw dataset, which is published on Figshare.

The Altmetric.com data includes counts of online attention at article level from across a variety of different data sources. In order to simplify data analysis, we aggregated data counts into the four classes as defined above: Social Activity, Mass Media, Scholarly Commentary, and Scholarly Activity. For each class, articles were assigned to predefined percentile ranges (cohorts) based on the frequency of online mentions (see Table 1).

Table 1 - Cohorts of articles based on the frequency of online attention within each class.

For example, the 69 papers comprising the top 0.5% of social activity comprise 91,470 social actions, 445 mass media mentions, 540 scholarly comments and 1,571 scholarly actions, whereas the top 69 papers comprising the top 0.5% of mass media activity comprise 2,638 mass media mentions, 16,221 social actions, 779 scholarly comments and 4,856 scholarly actions.

 Analysis

 Headline-grabbers: Which articles got most social media attention in 2013?

Of the 69 articles belonging to the 0.5% cohort in the Social Activity class (i.e. those articles most frequently mentioned in social media such as Twitter and Facebook, for example), just 8 of them are full-length articles reporting the results of original research. The remainder are typically editorial features or news items from leading weekly journals such as The Lancet, BMJ and Nature; see table 2 for the complete list. The original research articles cover topics in the popular consciousness including climate change, human health and diet, and online information and privacy: intuitively, the sort of articles one might expect to see attracting broad popular attention online. However, one article appears to have a less obvious popular slant (the Nature letter “Attractive photons in a quantum nonlinear medium”) but closer examination shows that it describes a novel technique for forcing photons to interact in a quantum nonlinear medium which may have applications in quantum processing, where the ability to have photons ‘see’ each other could overcome present technological limitations.

The remaining 61 articles (almost exclusively news and editorial features about original research reported elsewhere) cover a variety of topics including several on topics close to the heart of the academy: research careers, science funding, the future of higher education and scholarly publishing. The preponderance of items in this group from Nature (primarily the Nature News and Nature News Feature sections of the publication) suggest that Social activity may be more likely to reflect attention to short journalistic versions of current research results rather than the original research articles themselves; a worthy follow-up to this study would be to track the variation in performance across altmetric indicator classes of an original research article and the current awareness ‘news-worthy’ version of the same research.

Table 2 -  Full list of the 69 articles belonging to the 0.5% cohort in the Social Activity class including journal, article title, and DOI. Articles highlighted in orange are those representing full-length articles reporting the results of original research.

Social media attention: An indicator of scholarly impact or simply newsworthiness?

The articles which appear in the top 0.5% cohort in each of the four classes defined in this study are typically not the same ones: just 2 articles appear in all 4 lists. This suggests that the correlation between these 4 classes of altmetric indicators may not be very high. These two articles are both original research articles, one reporting the development of a method for creating human brain-like structures (called “cerebral organoids”) in cell culture and using these to study the basis of brain development and disease (Nature article “Cerebral organoids model human brain development and microcephaly”); the other correlating online behaviour (in this case, Facebook ‘likes’) with personal information such as sexual orientation, ethnicity and political views, to create a model to predict such traits based solely on Facebook activity (PNAS article “Private traits and attributes are predictable from digital records of human behavior”).

Further analysis of the overlap between the top 0.5% cohorts in each altmetric class is shown in Table 3: by far the greatest overlaps occur between the Mass media and Scholarly commentary classes, the lowest between Social activity and Mass media or Scholarly activity, and a moderate degree of overlap for the remaining pairwise combinations. Taken together, this suggests that - at least amongst this handful of articles receiving the most online attention – articles attracting a high degree of Social activity attract relatively little attention from the Mass media or from Scholarly activity and only a moderate degree of scholarly commentary. Conversely, there is a very high co-occurrence of articles receiving Mass media attention and Scholarly commentary. Taken together, these observations suggest that Social activity in particular is an indicator of a very different kind of online attention than the other three classes.

Table 3 -  Co-occurrence counts of articles comprising the top 0.5% of articles in each class, where n varies between classes owing to tied rankings at the 0.5% cutoff between 69 and 76.

Figure 1 shows how this correlation varies across all percentile cohorts for articles with Social activity. Note that approximately 90% of social activity is constrained to 15% of articles, which is a significantly more skewed distribution than that of citations across articles within a journal (where some 90% of citations are to 50% of the articles; (3)).  This implies a scarce attention economy in the Social activity spectrum, with many articles competing for a rare resource (reader attention). The only altmetric class with a distribution of attention across articles similar to that of citations across articles is Scholarly activity (which correlates very poorly with Social activity), where approximately 90% of Scholarly activity is represented by some 30-40% of articles (data not shown). The convergence of the curves in Figure 1 around the 15% cohort implies that at this point attention in all 4 classes is equally scarce, while in the cohorts above this point the only class showing a considerable degree of co-occurrence with Social activity is Scholarly commentary (also borne out by the Table 3 for the 0.5% cohort).

Figure 1 - Proportion of total activity per article across predefined percentile ranges (cohorts) for social activity.

Conclusions

It is clear from this exploratory work that altmetrics hold great promise as a source of data, indicators and insights about online attention, usage and impact of published research outputs. What is currently less certain is the underlying nature of what is being measured by current indicators represented within the four broad classes analysed here, and what can (and cannot) be read into them for the purposes of assigning credit or assessing research impact at the level of individual researchers, journals, institutions or countries.

What is strikingly clear from the qualitative analysis of the top 0.5% of papers for Social Activity is the lack of mentions of titles that have particularly titillating or eye-catching keywords: although most of the links are to summaries of research, rather than primary research articles themselves, they all contains serious scientific material.

On the basis of this preliminary study, we urge caution in characterizing all altmetric indicators in a similar way, as it is likely that different indicators may measure different types of online attention from different types of readers. This finding is similar to that reported by Priem, Piwowar and Hemminger in 2012 (4). We also suggest that careful delineation of document types (as long used for citation-based indicators) must be applied to correctly evaluate (for example) the relative social activity attracted by a news or editorial item versus an original research article; these values are likely to be the inverse of their usual relationship in citation terms. In short, in the excitement and promise of this burgeoning new field of Informetrics, we must be sure to ask ourselves: what is it that we are measuring, and why?

Acknowledgements

This paper would not be possible without the kind support of Euan Adie at Altmetric.com in providing access to these data for research purposes.

References

The data set this paper was  based on is available online: