Why Altmetric scores should never be used to measure the merit of scientific publications (or 'how to tweet your way to honour and glory')

Abstract

Because journal impact factors are widely recognized as a seriously flawed means of assessing the merit of a scientific paper (Seglen 1997), and because it takes time before it is known how well cited a scientific paper will become, there is a demand for metrics that can quantify a paper’s impact rapidly after publication. One prominent recent development is that of ‘altmetrics’ which capitalize on dissemination of the work via social media. The company ‘Altmetric’ provides an articlelevel score, presented within a multicoloured badge that quantifies the extent to which the work has been picked up by various social and other media outlets, including Twitter, Facebook and blogs. This score is placed prominently alongside the abstract of every paper published in the majority of the main ecological journals. Although the Altmetric company’s website cautions that one should not read too much into these scores without digging ‘deeper into the numbers and looking at the qualitative data underneath’, it also emphasizes that ‘Altmetrics are becoming widely used in academia, by individuals (as evidence of influence for promotion and tenure and in applying for grants), institutions (for benchmarking a university’s overall performance)’, and that the Altmetric badges (showcasing the scores) ‘provide a quick and easy way of showcasing the value of your publishing program to internal and external stakeholders, such as funding institutions and editorial boards’. Indeed, increasing numbers of researchers are making use of the Altmetric scores of their work in their CVs and applications for jobs and tenure, at least when they reflect favourably on the author. If Altmetric scores are to be used as a reliable indicator of the merit of a scientific publication, then it is critical that they cannot be gamed, and that they are entirely independent of the actions of the author postpublication. To test if this is the case, I conducted a simple analysis on the first 100 papers published in the journal Ecology in 2015. For each paper I noted the Altmetric score presented alongside the paper’s abstract. Because Altmetric scores for most papers are determined in large part by how many Twitter users ‘tweeted’ about the paper, I then examined the tweets for that paper and recorded whether or not the paper had been tweeted about by its own authors, i.e., from a Twitter account that the author has primary control over (such as their personal Twitter account, or lab-group Twitter account). This analysis reveals that publications which were tweeted about by their own authors had Altmetric scores of 3.3 times greater than did the others when mean values were considered, or 4.0 times greater when median values were used (Table 1). There are two possible explanations for this outcome. The first is that through tweeting about their own work, the authors generated publicity for it that greatly elevated its Altmetric score. While it is noted on the Altmetric website that they ‘count one person as only one source’, each of the author’s ‘followers’ (who are obviously likely to be favourably inclined to the author) that re-tweet the author’s tweet, as well as the followers’ followers, are presumably all regarded as independent sources. This suggests that authors that tweet about their own work will greatly elevate their Altmetric scores simply on the basis of re-tweets especially if they have lots of loyal followers. The second possible explanation is that authors who maintain a Twitter account and who tweet about their own work are also on average better researchers whose work is more worthy of high Altmetric scores. For this to hold, it requires that those scientists whose work has the greatest impact are