Making Sense of Citations

Abstract

This month, I would like to share a few personal thoughts about bibliometric indicators and specifically citations. As any scientist, publisher or journal editor will likely admit, the number of downloads, reads or citations associated with a journal publication are, for better or worse, ubiquitous metrics in modern-day scientific publishing. But what does a citation tell us? If an author cites a publication, they are simply making a declaration that a piece of work has relevance to their activities/interests and is worthy of comment. A citation makes no judgment on the “quality” of the cited work, but rather informs the reader that the prior study is worth inspection. That said, and to many, the number of citations does provide a measure of the relative “importance” or “impact” of an article to the wider community. My intention here is not to settle that argument, although I would say that broad-brush citation counting clearly fails to assess impact at the article level, ignoring the influence of the research field or time of publication, and that more nuanced metrics, such the relative citation ratio, (1) are far more instructive. Rather, I would like to recount an incident in my own research group. In the course of his studies, one of my graduate students realized that he needed an optical sensor for Pd²⁺ quantification. The sensor needed to be accessible, simple to implement, provide for good analytical sensitivities and detection limits and work in aqueous media. He performed a literature search and soon came across a number of optical sensors that on paper looked promising. One of these looked especially interesting, since it was based on measuring the fluorescence of a readily available coumarin laser dye. The authors claimed that their “turn-off” sensor was cheap, provided excellent (nM) detection limits, could sense Pd²⁺ in aqueous environments and could detect Pd²⁺ in live cells. The study had been published in a well-respected journal specializing in photophysical and photochemical research and had garnered over 20 citations within the four years since publication. All looked fine, so we decided to adopt the sensor and use it for the problem in hand. After a few weeks of testing and experimentation, we realized that the sensor might not be as useful as we had been led to believe. Through systematic reproduction of the experimental procedures reported in the original paper and a number of additional experiments, we came to the (correct) conclusion that the coumarin derivative was in fact not a fluorescence sensor for Pd²⁺ but was rather an extremely poor pH sensor able to operate over a restricted range of 1.5 pH units. This was clearly disappointing, but scientific research is rarely straightforward, and setbacks of this kind are not uncommon. What was far more worrisome was the fact that a number of the experimental procedures reported in the original paper were inaccurately or incompletely presented. This hindered our assessment of the sensor and meant that much effort was required to pinpoint earlier mistakes. This personal anecdote, rather than being an opportunistic diatribe, is intended to highlight the importance of providing an accurate and complete description of experimental methods used to generate the data presented in a scientific publication and the consequences of publishing inaccurate or erroneous findings. Fortunately for us, we developed an alternative Pd²⁺ sensor and additionally reported our “re-evaluation” of original work in the same peer-reviewed journal. However, this made me think more deeply about how we use the literature to inform and underpin contemporary science. The most obvious problem faced by all researchers, whatever their field of expertise, is the sheer number of peer-reviewed papers published each year. To give you some idea of the problem, over 2.8 million new papers were published and indexed by the Scopus and Web of Science databases in 2022: a number 47% higher than in 2016. (2) Even the most dedicated researcher would only be able to read a miniscule fraction of all papers relevant to their interests, so how should one prioritize and select which papers should be looked at and which should not? There is obviously no correct answer to this question, but for many, the strategy of choice will involve the use of scientific abstract and citation databases, such as Web of Science, Scopus, PubMed, SciFinder and The Lens, to find publications relevant to their area of interest. A citation index or database is simply an ordered register of cited articles along with a register of citing articles. Its utility lies in its ability to connect or associate scientific concepts and ideas. Put simply, if an author cites a previously published piece of work in their own paper, they have created an unambiguous link between their science and the prior work. Science citation indexing in its modern form was introduced by Eugene Garfield in the 1950s, with the primary goal of simplifying information retrieval, rather than identifying “important” or “impactful” publications. (3) Interestingly, a stated driver of his original science citation index was also to “eliminate the uncritical citation of fraudulent, incomplete, or obsolete data by making it possible for the conscientious scholar to be aware of criticisms of earlier papers”. Indeed, Garfield opines that “even if there were no other use for the citation index than that of minimizing the citation of poor data, the index would be well worth the effort”. This particular comment takes me back to my “palladium problem”. Perhaps, if I had looked more closely at the articles that cited the original paper, I would have uncovered concerns regarding the method and its sensing utility? So, having a spare hour, I did exactly this. Of course, this is one paper from many millions, but the results were instructive to me at least. In broad terms, almost all citations (to the original paper) appeared in the introductory section and simply stated that a Pd²⁺ sensor based on a coumarin dye had been reported. 80% made no comment on the quality (in terms of performance metrics) or utility of the work, 15% were self-citations by the authors, with only one paper providing comment on an aspect of the original data. Based on this analysis, I do not think that we can be too hard on ourselves for believing that the Pd²⁺ sensor would be fit for purpose. Nonetheless, how could we have leveraged the tools and features of modern electronic publishing to make a better analysis? One possible strategy could be to discriminate between citations based on their origin. For example, references in review articles may often have been cited without any meaningful analysis of the veracity of the work, while references cited in the results section of a research article are more likely to have been scrutinized by the authors in relation to their own work, whether the citation highlights a “good” or “bad” issue. Providing the reader with such information would clearly impart extra contrast to the citation metric and aid in their ability to identify articles “important” to their work. Fortunately, the advent of AI is beginning to make valuable contributions in this regard and a number of “smart citation” tools are being introduced. For example, citation analysis platforms such as Scite (4) leverage AI to better understand and utilize scientific citations. Rather than simply reporting the occurrence of a citation, citations can be classified by their contextual usage, for example, through the number of supporting, contrasting, and mentioning citation statements. This allows researchers to evaluate the utility and importance of a reference and ultimately enhance the scientific method. This would be especially useful in our field of sensor science, where knowledge of the sensors or sensing methods that have been successfully used in given scenarios would be invaluable when identifying the need to improve or develop new sensors. It will be some time before “smart citation metrics” are widely adopted by the scientific community. However, it is clear that all citations are not equal, and that we should be smarter in both the way we cite literature and the way we use literature citations. This article references 4 other publications. This article has not yet been cited by other publications.