
Party papers or policy discussions: an examination of highly shared papers using altmetric data
Which scientific stories are most shared on social media networks such as Twitter and Facebook?
Research on human health and social issues are often perceived as being the most shared scientific stories on social media networks such as Twitter and Facebook and – given their mainstream appeal – are often suggested to dominate the popular discussion around scholarly research online, but skeptics, such as David Calquhoun argue for their irrelevance: “Scientific works get tweeted about mostly because they have titles that contain buzzwords, not because they represent great science” (1).
So which is it to be? And do articles attracting social media attention also get the attention of scholars and the mass media? In this article, we seek to provide an approach to answering these questions.
With the rise of online scholarly publishing and the concomitant rise in the desire to create indicators of online attention to research articles and related outputs have come a number of providers of article-level data. A leading commercial provider of such data - collectively known as ‘altmetrics’ - is Altmetric.com, which tracks a variety of different indicators in four broad groups: Social Activity (e.g. Tweets and Facebook mentions), Mass Media (e.g. mentions on news sites such as BBC and CNN), Scholarly Commentary (e.g. mentions in scientific blogs), and Scholarly Activity (e.g. articles in reader libraries such as Mendeley). The overall collection and analysis of these references are brought together under the label “altmetrics”.
In terms of the volume of online mentions of scholarly articles, Twitter and other social networks provide by far the largest number of data points. However, given Twitter’s broad user base (the majority being non-academics) and limited information content (being restricted to 140 characters per tweet), other indicators may be more significant in terms of understanding scholarly usage (2). For example, Mendeley and CiteULike are examples of sharing and collaboration platforms used predominantly by researchers, while the mass media and scientific blogs tracked by Altmetric.com are written by professional science journalists or researchers themselves.
Methodology
Data were collected from the Altmetric.com API over four months ending January 17th, 2014. On this date, the latest altmetric indicator data for all papers published in a selection of journals in 2013 with any online mentions captured by Altmetric.com were downloaded for analysis; in total, 13,793 articles with at least one altmetric indicator datapoint were included in this study. Please note, the actual Journals monitored are detailed in the raw dataset, which is published on Figshare.
The Altmetric.com data includes counts of online attention at article level from across a variety of different data sources. In order to simplify data analysis, we aggregated data counts into the four classes as defined above: Social Activity, Mass Media, Scholarly Commentary, and Scholarly Activity. For each class, articles were assigned to predefined percentile ranges (cohorts) based on the frequency of online mentions (see Table 1).
Cohorts | Number of articles included |
0.5% |
69 |
1% |
138 |
2.5% |
347 |
5% |
691 |
7.5% |
1,045 |
10% |
1,384 |
15% |
2,095 |
20% |
2,775 |
25% |
3,539 |
30% |
4,332 |
100% |
13,793 |
Table 1 - Cohorts of articles based on the frequency of online attention within each class.
For example, the 69 papers comprising the top 0.5% of social activity comprise 91,470 social actions, 445 mass media mentions, 540 scholarly comments and 1,571 scholarly actions, whereas the top 69 papers comprising the top 0.5% of mass media activity comprise 2,638 mass media mentions, 16,221 social actions, 779 scholarly comments and 4,856 scholarly actions.
Analysis
Headline-grabbers: Which articles got most social media attention in 2013?
Of the 69 articles belonging to the 0.5% cohort in the Social Activity class (i.e. those articles most frequently mentioned in social media such as Twitter and Facebook, for example), just 8 of them are full-length articles reporting the results of original research. The remainder are typically editorial features or news items from leading weekly journals such as The Lancet, BMJ and Nature; see table 2 for the complete list. The original research articles cover topics in the popular consciousness including climate change, human health and diet, and online information and privacy: intuitively, the sort of articles one might expect to see attracting broad popular attention online. However, one article appears to have a less obvious popular slant (the Nature letter “Attractive photons in a quantum nonlinear medium”) but closer examination shows that it describes a novel technique for forcing photons to interact in a quantum nonlinear medium which may have applications in quantum processing, where the ability to have photons ‘see’ each other could overcome present technological limitations.
The remaining 61 articles (almost exclusively news and editorial features about original research reported elsewhere) cover a variety of topics including several on topics close to the heart of the academy: research careers, science funding, the future of higher education and scholarly publishing. The preponderance of items in this group from Nature (primarily the Nature News and Nature News Feature sections of the publication) suggest that Social activity may be more likely to reflect attention to short journalistic versions of current research results rather than the original research articles themselves; a worthy follow-up to this study would be to track the variation in performance across altmetric indicator classes of an original research article and the current awareness ‘news-worthy’ version of the same research.
Journal | Article title | DOI |
Nature | Cerebral organoids model human brain development and microcephaly | 10.1038/nature12517 |
Nature Comment | Climate science: Vast costs of Arctic change | 10.1038/499401a |
Nature Comment | Neuroscience: My life with Parkinson's | 10.1038/503029a |
Nature Editorial | Nuclear error | 10.1038/501005b |
Nature Editorial | Science for all | 10.1038/495005a |
Nature Letter | No increase in global temperature variability despite changing regional patterns | 10.1038/nature12310 |
Nature Letter | Attractive photons in a quantum nonlinear medium | 10.1038/nature12512 |
Nature News | Brazilian citation scheme outed | 10.1038/500510a |
Nature News | Half of 2011 papers now free to read | 10.1038/500386a |
Nature News | World's slowest-moving drop caught on camera at last | 10.1038/nature.2013.13418 |
Nature News | Genetically modified crops pass benefits to weeds | 10.1038/nature.2013.13517 |
Nature News | NSF cancels political-science grant cycle | 10.1038/nature.2013.13501 |
Nature News | Deal done over HeLa cell line | 10.1038/500132a |
Nature News | Antibiotic resistance: The last resort | 10.1038/499394a |
Nature News | Cosmologist claims Universe may not be expanding | 10.1038/nature.2013.13379 |
Nature News | Zapped malaria parasite raises vaccine hopes | 10.1038/nature.2013.13536 |
Nature News | See-through brains clarify connections | 10.1038/496151a |
Nature News | Dolphins remember each other for decades | 10.1038/nature.2013.13519 |
Nature News | Researchers turn off Down’s syndrome genes | 10.1038/nature.2013.13406 |
Nature News | Astrophysics: Fire in the hole! | 10.1038/496020a |
Nature News | Giant viruses open Pandora's box | 10.1038/nature.2013.13410 |
Nature News | Quantum gas goes below absolute zero | 10.1038/nature.2013.12146 |
Nature News | Stem cells reprogrammed using chemicals alone | 10.1038/nature.2013.13416 |
Nature News | Whole human brain mapped in 3D | 10.1038/nature.2013.13245 |
Nature News | Father’s genetic quest pays off | 10.1038/498418a |
Nature News | Tracking whole colonies shows ants make career moves | 10.1038/nature.2013.12833 |
Nature News | Pesticides spark broad biodiversity loss | 10.1038/nature.2013.13214 |
Nature News | Animal-rights activists wreak havoc in Milan laboratory | 10.1038/nature.2013.12847 |
Nature News | Silver makes antibiotics thousands of times more effective | 10.1038/nature.2013.13232 |
Nature News | Methane leaks erode green credentials of natural gas | 10.1038/493012a |
Nature News | When Google got flu wrong | 10.1038/494155a |
Nature News | First proof that prime numbers pair up into infinity | 10.1038/nature.2013.12989 |
Nature News | Global carbon dioxide levels near worrisome milestone | 10.1038/497013a |
Nature News | Underwater volcano is Earth's biggest | 10.1038/nature.2013.13680 |
Nature News | Did a hyper-black hole spawn the Universe? | 10.1038/nature.2013.13743 |
PNAS | Private traits and attributes are predictable from digital records of human behavior | 10.1073/pnas.1218772110 |
Nature News | How to turn living cells into computers | 10.1038/nature.2013.12406 |
Nature News | Small-molecule drug drives cancer cells to suicide | 10.1038/nature.2013.12385 |
Nature News | Brain-simulation and graphene projects win billion-euro competition | 10.1038/nature.2013.12291 |
Nature News | Rewired nerves control robotic leg | 10.1038/nature.2013.13818 |
Nature News | US government shuts down | 10.1038/502013a |
Lancet Letter | Open letter: let us treat patients in Syria | 10.1016/s0140-6736(13)61938-8 |
Nature News | Blood engorged mosquito is a fossil first | 10.1038/nature.2013.13946 |
BMJ | Cancer risk in 680 000 people exposed to computed tomography scans in childhood or adolescence: data linkage study of 11 million Australians | 10.1136/bmj.f2360 |
Nature News | NIH mulls rules for validating key results | 10.1038/500014a |
PNAS | Impact of insufficient sleep on total daily energy expenditure, food intake, and weight gain | 10.1073/pnas.1216951110 |
Nature News | Red meat + wrong bacteria = bad news for hearts | 10.1038/nature.2013.12746 |
Nature News | Who is the best scientist of them all? | 10.1038/nature.2013.14108 |
Nature News | Four-strand DNA structure found in cells | 10.1038/nature.2013.12253 |
Nature News | Weak statistical standards implicated in scientific irreproducibility | 10.1038/nature.2013.14131 |
Nature News | Mathematicians aim to take publishers out of publishing | 10.1038/nature.2013.12243 |
BMJ | Bicycle helmets and the law | 10.1136/bmj.f3817 |
Nature News | Barbaric Ostrich: 27th June 2013 | 10.1038/nature.2013.12487 |
American J of M | The Autopsy of Chicken Nuggets Reads “Chicken Little” | 10.1016/j.amjmed.2013.05.005 |
Nature News | Stem cells mimic human brain | 10.1038/nature.2013.13617 |
Nature News | Mystery humans spiced up ancients’ sex lives | 10.1038/nature.2013.14196 |
BMJ | The future of the NHS--irreversible privatisation? | 10.1136/bmj.f1848 |
Nature News Feature | Archaeology: The milk revolution | 10.1038/500020a |
Nature News Feature | Neuroscience: Solving the brain | 10.1038/499272a |
Nature News Feature | Tissue engineering: How to build a heart | 10.1038/499020a |
Nature News Feature | Theoretical physics: The origins of space and time | 10.1038/500516a |
Nature News Feature | Online learning: Campus 2.0 | 10.1038/495160a |
Nature News Feature | Open access: The true cost of science publishing | 10.1038/495426a |
Nature News Feature | Inequality quantified: Mind the gender gap | 10.1038/495022a |
Nature News Feature | Voyager: Outward bound | 10.1038/497424a |
Nature News Feature | Mental health: On the spectrum | 10.1038/496416a |
Nature News Feature | Brain decoding: Reading minds | 10.1038/502428a |
Nature News Feature | Fukushima: Fallout of fear | 10.1038/493290a |
Nature News Feautre | The big fat truth | 10.1038/497428a |
Table 2 - Full list of the 69 articles belonging to the 0.5% cohort in the Social Activity class including journal, article title, and DOI. Articles highlighted in orange are those representing full-length articles reporting the results of original research.
Social media attention: An indicator of scholarly impact or simply newsworthiness?
The articles which appear in the top 0.5% cohort in each of the four classes defined in this study are typically not the same ones: just 2 articles appear in all 4 lists. This suggests that the correlation between these 4 classes of altmetric indicators may not be very high. These two articles are both original research articles, one reporting the development of a method for creating human brain-like structures (called “cerebral organoids”) in cell culture and using these to study the basis of brain development and disease (Nature article “Cerebral organoids model human brain development and microcephaly”); the other correlating online behaviour (in this case, Facebook ‘likes’) with personal information such as sexual orientation, ethnicity and political views, to create a model to predict such traits based solely on Facebook activity (PNAS article “Private traits and attributes are predictable from digital records of human behavior”).
Further analysis of the overlap between the top 0.5% cohorts in each altmetric class is shown in Table 3: by far the greatest overlaps occur between the Mass media and Scholarly commentary classes, the lowest between Social activity and Mass media or Scholarly activity, and a moderate degree of overlap for the remaining pairwise combinations. Taken together, this suggests that - at least amongst this handful of articles receiving the most online attention – articles attracting a high degree of Social activity attract relatively little attention from the Mass media or from Scholarly activity and only a moderate degree of scholarly commentary. Conversely, there is a very high co-occurrence of articles receiving Mass media attention and Scholarly commentary. Taken together, these observations suggest that Social activity in particular is an indicator of a very different kind of online attention than the other three classes.
Mass media | Scholarly activity | Scholarly commentary | Social activity | |
Mass media |
11 |
31 |
5 |
|
Scholarly activity |
14 |
2 |
||
Scholarly commentary |
15 |
|||
Social activity |
Table 3 - Co-occurrence counts of articles comprising the top 0.5% of articles in each class, where n varies between classes owing to tied rankings at the 0.5% cutoff between 69 and 76.
Figure 1 shows how this correlation varies across all percentile cohorts for articles with Social activity. Note that approximately 90% of social activity is constrained to 15% of articles, which is a significantly more skewed distribution than that of citations across articles within a journal (where some 90% of citations are to 50% of the articles; (3)). This implies a scarce attention economy in the Social activity spectrum, with many articles competing for a rare resource (reader attention). The only altmetric class with a distribution of attention across articles similar to that of citations across articles is Scholarly activity (which correlates very poorly with Social activity), where approximately 90% of Scholarly activity is represented by some 30-40% of articles (data not shown). The convergence of the curves in Figure 1 around the 15% cohort implies that at this point attention in all 4 classes is equally scarce, while in the cohorts above this point the only class showing a considerable degree of co-occurrence with Social activity is Scholarly commentary (also borne out by the Table 3 for the 0.5% cohort).
Figure 1 - Proportion of total activity per article across predefined percentile ranges (cohorts) for social activity.
Conclusions
It is clear from this exploratory work that altmetrics hold great promise as a source of data, indicators and insights about online attention, usage and impact of published research outputs. What is currently less certain is the underlying nature of what is being measured by current indicators represented within the four broad classes analysed here, and what can (and cannot) be read into them for the purposes of assigning credit or assessing research impact at the level of individual researchers, journals, institutions or countries.
What is strikingly clear from the qualitative analysis of the top 0.5% of papers for Social Activity is the lack of mentions of titles that have particularly titillating or eye-catching keywords: although most of the links are to summaries of research, rather than primary research articles themselves, they all contains serious scientific material.
On the basis of this preliminary study, we urge caution in characterizing all altmetric indicators in a similar way, as it is likely that different indicators may measure different types of online attention from different types of readers. This finding is similar to that reported by Priem, Piwowar and Hemminger in 2012 (4). We also suggest that careful delineation of document types (as long used for citation-based indicators) must be applied to correctly evaluate (for example) the relative social activity attracted by a news or editorial item versus an original research article; these values are likely to be the inverse of their usual relationship in citation terms. In short, in the excitement and promise of this burgeoning new field of Informetrics, we must be sure to ask ourselves: what is it that we are measuring, and why?
Acknowledgements
This paper would not be possible without the kind support of Euan Adie at Altmetric.com in providing access to these data for research purposes.
References
(1) http://www.dcscience.net/?p=6369 (2) http://www.slideshare.net/StefanieHaustein/haustein-ape2014-30482551 (3) Seglen, P.O. (1992) The skewness of science. Journal of the American Society for Information Science, 43 (9) pp. 628–638. (4) Priem, J., Piwowar, H., & Hemminger, B. (2012) Altmetrics in the wild: Using social media to explore scholarly impact. Arxiv. http://arxiv.org/abs/1203.4745The data set this paper was based on is available online:
Taylor, Michael (2014): Data set for " Party papers or policy discussions: an examination of highly shared papers using altmetric data". figshare. http://dx.doi.org/10.6084/m9.figshare.943471